Skip to main content
SpringerLink
Log in

Curating GitHub for engineered software projects

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software forges like GitHub host millions of repositories. Software engineering researchers have been able to take advantage of such a large corpora of potential study subjects with the help of tools like GHTorrent and Boa. However, the simplicity in querying comes with a caveat: there are limited means of separating the signal (e.g. repositories containing engineered software projects) from the noise (e.g. repositories containing home work assignments). The proportion of noise in a random sample of repositories could skew the study and may lead to researchers reaching unrealistic, potentially inaccurate, conclusions. We argue that it is imperative to have the ability to sieve out the noise in such large repository forges. We propose a framework, and present a reference implementation of the framework as a tool called reaper, to enable researchers to select GitHub repositories that contain evidence of an engineered software project. We identify software engineering practices (called dimensions) and propose means for validating their existence in a GitHub repository. We used reaper to measure the dimensions of 1,857,423 GitHub repositories. We then used manually classified data sets of repositories to train classifiers capable of predicting if a given GitHub repository contains an engineered software project. The performance of the classifiers was evaluated using a set of 200 repositories with known ground truth classification. We also compared the performance of the classifiers to other approaches to classification (e.g. number of GitHub Stargazers) and found our classifiers to outperform existing approaches. We found stargazers-based classifier (with 10 as the threshold for number of stargazers) to exhibit high precision (97%) but an inversely proportional recall (32%). On the other hand, our best classifier exhibited a high precision (82%) and a high recall (86%). The stargazer-based criteria offers precision but fails to recall a significant portion of the population.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. National Science Foundation (NSF) Grant CNS-1513263

  2. https://github.com/about/press

  3. https://github.com/RepoReapers/reaper/wiki/Extending-reaper

  4. http://doc.qt.io/qt-5/qtgraphicaleffects-index.html

  5. http://merproject.org/

  6. Citation counts retrieved from Google Scholar

References

  • Allamanis M, Sutton C (2013) Mining source code repositories at massive scale using language modeling Proceedings of the 10th working conference on mining software repositories, IEEE Press, Piscataway, NJ, USA, MSR ’13, pp 207–216. http://dl.acm.org/citation.cfm?id=2487085.2487127

  • Belady LA, Lehman MM (1976) A model of large program development. IBM Syst J 15(3):225–252. doi:10.1147/sj.153.0225

    Article  MATH  Google Scholar 

  • Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code!: examining the effects of ownership on software quality Proceedings of the 19th ACM SIGSOFT symposium and the 13th european conference on foundations of software engineering, ACM, New York, NY, USA, ESEC/FSE ’11, pp 4–14. doi:10.1145/2025113.2025119

  • Bissyandé TF, Lo D, Jiang L, Réveillère L, Klein J, Traon YL (2013) Got issues? Who cares about it? A large scale investigation of issue trackers from GitHub 2013 IEEE 24th international symposium on software reliability engineering (ISSRE), pp 188–197. doi:10.1109/ISSRE.2013.6698918

  • Bissyandé TF, Thung F, Lo D, Jiang L, Réveillère L (2013a) Orion: a software project search engine with integrated diverse software artifacts 2013 18th international conference on engineering of complex computer systems, pp 242–245. doi:10.1109/ICECCS.2013.42

  • Bissyandé TF, Thung F, Lo D, Jiang L, Réveillère L (2013b) Popularity, interoperability, and impact of programming languages in 100,000 open source projects 2013 IEEE 37th annual computer software and applications conference, pp 303–312. doi:10.1109/COMPSAC.2013.55

  • Breiman L (2001) Random forests. Mach Learn 45(1):5–32. doi:10.1023/A:1010933404324

    Article  MATH  Google Scholar 

  • CA Technologies (2016) Waffle.io - work better on GitHub issues. https://waffle.io/, accessed: 2016-03-11

  • Carlo Z (2016) Githut - programming languages and GitHub. http://githut.info, accessed: 2016-03-11

  • Codetree Studios (2016) Codetree - GitHub issues, managed. https://codetree.com/, accessed: 2016-03-11

  • Danial A (2014) CLOC – Count lines of code. http://cloc.sourceforge.net/, accessed: 2016-03-11, version: 1.62

  • de Souza CB, Anquetil N, de Oliveira KM (2005) A study of the documentation essential to software maintenance Proceedings of the 23rd annual international conference on design of communication: documenting & designing for pervasive information, ACM, New York, NY, USA, SIGDOC ’05, pp 68–75. doi:10.1145/1085313.1085331

  • Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: a language and infrastructure for analyzing ultra-large-scale software repositories Proceedings of the 2013 international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’13, pp 422–431. http://dl.acm.org/citation.cfm?id=2486788.2486844

  • Eick SG, Graves TL, Karr AF, Marron JS, Mockus A (2001) Does code decay? Assessing the evidence from change management data. IEEE Trans Softw Eng 27(1):1–12. doi:10.1109/32.895984

    Article  Google Scholar 

  • Emam KE, Benlarbi S, Goel N, Rai SN (2001) The confounding effect of class size on the validity of object-oriented metrics. IEEE Trans Softw Eng 27(7):630–650. doi:10.1109/32.935855

    Article  Google Scholar 

  • GHTorrent (2016a) Hall of fame. http://ghtorrent.org/halloffame.html, accessed: 2016-03-11

  • GHTorrent (2016b) The relational DB schema. http://ghtorrent.org/relational.html, accessed: 2016-03-11

  • GitHub Inc (2016a) Github API v3—github developer guide. https://developer.github.com/v3/, accessed: 2016-03-11

  • GitHub Inc (2016b) Github archive. https://www.githubarchive.org/, accessed: 2016-06-19

  • GitHub Inc (2016c) No license - choose a license. http://choosealicense.com/no-license/, accessed: 2016-03-11

  • Gousios G (2013) The GHTorrent dataset and tool suite Proceedings of the 10th working conference on mining software repositories, IEEE Press, Piscataway, NJ, USA, MSR ’13, pp 233–236. http://dl.acm.org/citation.cfm?id=2487085.2487132

  • Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in GitHub: an empirical study Proceedings of the 11th working conference on mining software repositories, ACM, New York, NY, USA, MSR 2014, pp 352–355. doi:10.1145/2597073.2597118

  • HuBoard Inc (2016) Huboard - github issues made awesome. https://huboard.com/, accessed: 2016-03-11

  • Iowa State University (2016) Publications related to Boa - Boa - Iowa State University. http://boa.cs.iastate.edu/papers/, accessed: 2016-03-11

  • Jarczyk O, Gruszka B, Jaroszewicz S, Bukowski L, Wierzbicki A (2014) Github projects. Quality analysis of open-source software. Springer International Publishing, Cham, pp 80–94. doi:10.1007/978-3-319-13734-6_6

    Google Scholar 

  • Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub Proceedings of the 11th working conference on mining software repositories, ACM, New York, NY, USA, MSR 2014, pp 92–101. doi:10.1145/2597073.2597074

  • Kochhar PS, Bissyandé TF, Lo D, Jiang L (2013) Adoption of software testing in open source projects–a preliminary study on 50,000 projects 2013 17th european conference on software maintenance and reengineering, pp 353–356. doi:10.1109/CSMR.2013.48

  • Kofink A (2015) Contributions of the under-appreciated: gender bias in an open-source ecology Companion proceedings of the 2015 ACM SIGPLAN international conference on systems, programming, languages and applications: Software for humanity, ACM, New York, NY, USA, SPLASH Companion 2015, pp 83–84. doi:10.1145/2814189.2815369

  • Laplante P (2007) What every engineer should know about software engineering. What every engineer should know. CRC Press

  • Mockus A, Fielding R T, Herbsleb J (2000) A case study of open source software development: the apache server Proceedings of the 2000 international conference on software engineering. ICSE 2000 the new millennium, pp 263–272. doi:10.1145/337180.337209

  • Munaiah N, Kroh S, Cabrey C, Nagappan M (2016a) Home of the reporeapers. https://reporeapers.github.io, accessed: 2016-03-11

  • Munaiah N, Kroh S, Cabrey C, Parikh N (2016b) Reaper - reference implementation. https://github.com/reporeapers/reaper, accessed: 2016-03-11

  • Nagappan N (2007) Potential of open source systems as project repositories for empirical studies working group results. Springer, Berlin, pp 103–107. doi:10.1007/978-3-540-71301-2_29

    Google Scholar 

  • Nagappan N, Williams L, Osborne J, Vouk M, Abrahamsson P (2005) Providing test quality feedback using static source code and automatic test suite metrics 16th IEEE international symposium on software reliability engineering (ISSRE’05), pp 10–94. doi:10.1109/ISSRE.2005.35

  • Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, FSE 2014, pp 155–165. doi:10.1145/2635868.2635922

  • Rosenberg J (1997) Some misconceptions about lines of code Proceedings fourth international software metrics symposium, pp 137–142. doi:10.1109/METRIC.1997.637174

  • Ross SM (2003) Peirce’s criterion for the elimination of suspect experimental data. J Eng Technol 20(2):38–41

    Google Scholar 

  • Sajnani H, Saini V, Ossher J, Lopes CV (2014) Is popularity a measure of quality? an analysis of maven components 2014 IEEE international conference on software maintenance and evolution, pp 231–240. doi:10.1109/ICSME.2014.45

  • Software Freedom Law Center (2012) Managing copyright information within a free software project - software freedom law center. http://softwarefreedom.org/resources/2012/managingcopyrightinformation.html, accessed: 2015-05-15

  • Syer MD, Nagappan M, Hassan AE, Adams B (2013) Revisiting prior empirical findings for mobile apps: an empirical case study on the 15 most popular open-source android apps Proceedings of the 2013 conference of the center for advanced studies on collaborative research, IBM corp., riverton, NJ, USA, CASCON ’13, pp 283–297. http://dl.acm.org/citation.cfm?id=2555523.2555553

  • Tung YH, Chuang CJ, Shan HL (2014) A framework of code reuse in open source software The 16th asia-pacific network operations and management symposium, pp 1–6. doi:10.1109/APNOMS.2014.6996525

  • Vasilescu B, van Schuylenburg S, Wulms J, Serebrenik A, van den Brand MGJ (2014) Continuous integration in a social-coding world empirical evidence from GitHub 2014 IEEE international conference on software maintenance and evolution, pp 401–405. doi:10.1109/ICSME.2014.62

  • Vendome C (2015) A large scale study of license usage on GitHub 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2, pp 772–774. doi:10.1109/ICSE.2015.245

  • Whitehead J, Mistrík I, Grundy J, van der Hoek A (2010) Collaborative software engineering: concepts and techniques. Springer, Berlin, pp 1–30. doi:10.1007/978-3-642-10294-3_1

    Book  MATH  Google Scholar 

  • Zaidman A, Rompaey BV, Demeyer S, v Deursen A (2008) Mining software repositories to study co-evolution of production & test code 2008 1st international conference on software testing, verification, and validation, pp 220–229. doi:10.1109/ICST.2008.47

  • Zenhub (2016) Zenhub - project management for agile teams on GitHub. https://www.zenhub.io/, accessed: 2016-03-11

  • Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366–427. doi:10.1145/267580.267590

    Article  Google Scholar 

Download references

Acknowledgments

We thank Nimish Parikh for his contributions during the initial stages of the research. We thank our peers for providing us their GitHub authentication keys which helped us attain the API bandwidth required. We also thank the Research Computing Group at Rochester Institute of Technology for providing us the computing resources necessary to execute a project of this scale.

Author information

Authors and Affiliations

  1. Department of Software Engineering, Rochester Institute of Technology, 134 Lomb Memorial Drive, Rochester, NY, 14623, USA

    Nuthan Munaiah, Steven Kroh & Craig Cabrey

  2. David R. Cheriton School of Computer Science, University of Waterloo, 200 University Avenue West, Waterloo, Ontario, N2L 3G1, Canada

    Meiyappan Nagappan

Authors
  1. Nuthan Munaiah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven Kroh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Craig Cabrey
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Meiyappan Nagappan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nuthan Munaiah.

Additional information

Communicated by: Denys Poshyvanyk

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Munaiah, N., Kroh, S., Cabrey, C. et al. Curating GitHub for engineered software projects. Empir Software Eng 22, 3219–3253 (2017). https://doi.org/10.1007/s10664-017-9512-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9512-6

Keywords

Search

Navigation