Skip to main content
SpringerLink
Log in

Studying the relationship between logging characteristics and the code quality of platform software

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Platform software plays an important role in speeding up the development of large scale applications. Such platforms provide functionalities and abstraction on which applications can be rapidly developed and easily deployed. Hadoop and JBoss are examples of popular open source platform software. Such platform software generate logs to assist operators in monitoring the applications that run on them. These logs capture the doubts, concerns, and needs of developers and operators of platform software. We believe that such logs can be used to better understand code quality. However, logging characteristics and their relation to quality has never been explored. In this paper, we sought to empirically study this relation through a case study on four releases of Hadoop and JBoss. Our findings show that files with logging statements have higher post-release defect densities than those without logging statements in 7 out of 8 studied releases. Inspired by prior studies on code quality, we defined log-related product metrics, such as the number of log lines in a file, and log-related process metrics such as the number of changed log lines. We find that the correlations between our log-related metrics and post-release defects are as strong as their correlations with traditional process metrics, such as the number of pre-release defects, which is known to be one the metrics with the strongest correlation with post-release defects. We also find that log-related metrics can complement traditional product and process metrics resulting in up to 40 % improvement in explanatory power of defect proneness. Our results show that logging characteristics provide strong indicators of defect-prone source code files. However, we note that removing logs is not the answer to better code quality. Instead, our results show that it might be the case that developers often relay their concerns about a piece of code through logs. Hence, code quality improvement efforts (e.g., testing and inspection) should focus more on the source code files with large amounts of logs or with large amounts of log churn.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. https://issues.apache.org/jira/browse/HADOOP-2490 last checked on May 2013.

  2. https://issues.apache.org/jira/browse/HADOOP-1034 last checked on May 2013.

  3. http://sosp2011.gsd.inesc-id.pt/workshops/slaml, last checked on December 2012.

  4. http://wiki.eclipse.org/Mylyn last checked on May 2013.

References

  • Bacchelli A, D’Ambros M, Lanza M (2010) Are popular classes more defect prone? In: FASE ’10: proceedings of the 13th international conference on fundamental approaches to software engineering. Springer, Berlin, pp 59–73

  • Barbour L, Khomh F, Zou Y (2011) Late propagation in software clones. In: ICSM 2011: proceedings of the 27th IEEE international conference on software maintenance, pp 273–282

  • Beschastnikh I, Brun Y, Schneider S, Sloan M, Ernst MD (2011) Leveraging existing instrumentation to automatically infer invariant-constrained models. In: ESEC/FSE ’11: proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering. ACM, New York, pp 267–277

  • Bettenburg N, Hassan A (2013) Studying the impact of social interactions on software quality. Empir Software Eng 18(2):375–431

    Article  Google Scholar 

  • Bettenburg N, Hassan AE (2010) Studying the impact of social structures on software quality. In: ICPC ’10: proceedings of the 18th international conference on program comprehension. IEEE Computer Society, Washington, pp 124–133

  • Bird C, Bachmann A, Aune E, Duffy J, Bernstein A, Filkov V, Devanbu P (2009) Fair and balanced?: bias in bug-fix datasets. In: ESEC/FSE ’09: proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering. ACM, New York, pp 121–130

  • Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  • Cataldo M, Mockus A, Roberts JA, Herbsleb JD (2009) Software dependencies, work dependencies, and their impact on failures. IEEE Trans Softw Eng 35:864–878

    Article  Google Scholar 

  • Chen TH, Thomas SW, Nagappan M, Hassan AE (2012) Explaining software defects using topic models. In: MSR. IEEE, pp 189–198

  • Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113

    Article  Google Scholar 

  • Fenton N, Neil M (1999) A critique of software defect prediction models. IEEE Trans Softw Eng 25(5):675–689

    Article  Google Scholar 

  • Gilstrap BR (2002) An introduction to the java logging api. http://www.onjava.com/pub/a/onjava/2002/06/19/log.html

  • Graves TL, Karr AF, Marron J, Siy H (2000) Predicting fault incidence using software change history. IEEE Trans Softw Eng 26(7):653–661

    Article  Google Scholar 

  • Graves TL, Karr AF, Marron JS, Siy H (2000) Predicting fault incidence using software change history. IEEE Trans Softw Eng 26:653–661

    Article  Google Scholar 

  • Hall T, Beecham S, Bowes D, Gray D, Counsell S (2012) A systematic literature review on fault prediction performance in software engineering. IEEE Trans Softw Eng 38(6):1276–1304

    Article  Google Scholar 

  • Harrell F (2001) Regression modeling strategies with applications to linear models, logistic regression, and survival analysis. Springer

  • Hassan AE (2005) Mining software repositories to assist developers and support managers. Ph.D. thesis, University of Waterloo

  • Hassan AE (2008) Automated classification of change messages in open source projects. In: SAC ’08: proceedings of the 2008 ACM symposium on applied computing. ACM, New York, pp 837–841

  • Hassan AE (2009) Predicting faults using the complexity of code changes. In: ICSE ’09: proceedings of the 2009 IEEE 31st international conference on software engineering. IEEE Computer Society, Washington, pp 78–88

  • Herraiz I, Hassan A (2010) Beyond lines of code: do we need more complexity metrics? In: Oram A, Wilson G (eds) Making software: what really works, and why we believe it? OReilly Media

  • Ibrahim WM, Bettenburg N, Adams B, Hassan AE (2012) On the relationship between comment update practices and software bugs. J Syst Softw 85(10):2293–2304

    Article  Google Scholar 

  • Ihaka R, Gentleman R (1996) R: a language for data analysis and graphics. J Comput Graph Stat 5(3):299–314

    Google Scholar 

  • Jackson J, Wiley J (1991) A user’s guide to principal components, vol 19. Wiley Online Library

  • Jiang W, Hu C, Pasupathy S, Kanevsky A, Li Z, Zhou Y (2009) Understanding customer problem troubleshooting from storage system logs. In: FAST ’09: proccedings of the 7th conference on file and storage technologies. USENIX Association, Berkeley, pp 43–56

  • Jiang ZM, Hassan AE, Hamann G, Flora P (2008) Automatic identification of load testing problems. In: ICSM ’08: proc. of 24th IEEE international conference on software maintenance. IEEE, Beijing, pp 307–316

  • Jiang ZM, Hassan AE, Hamann G, Flora P (2009) Automated performance analysis of load tests. In: ICSM ’09: proc. of the 25th IEEE international conference on software maintenance. IEEE, Edmonton, pp 125–134

  • Liu Y, Poshyvanyk D, Ferenc R, Gyimothy T, Chrisochoides N (2009) Modeling class cohesion as mixtures of latent topics. In: ICSM 2009: proceedings of the 2009 IEEE international conference on software maintenance, pp 233–242

  • Meneely A, Williams L, Snipes W, Osborne J (2008) Predicting failures with developer networks and social network analysis. In: SIGSOFT ’08/FSE-16: proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering, SIGSOFT ’08/FSE-16. ACM, New York, pp 13–23

  • Menzies T, Greenwald J, Frank A (2007) Data mining static code attributes to learn defect predictors. IEEE Trans Softw Eng 33(1):2–13

    Article  Google Scholar 

  • Mockus A (2010) Organizational volatility and its effects on software defects. In: FSE ’10: proc. of the 18th ACM SIGSOFT international symp. on foundations of software engineering. ACM, New York, pp 117–126

  • Mockus A, Votta LG (2000) Identifying reasons for software changes using historic databases. In: ICSM ’00: proceedings of the international conference on software maintenance. IEEE Computer Society, Washington, pp 120–130

  • Mockus A, Weiss DM (2000) Predicting risk of software changes. Bell Labs Tech J 5:169–180

    Article  Google Scholar 

  • Moser R, Pedrycz W, Succi G (2008) A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In: ICSE 2008: proceedings of the 30th international conference on software engineering. ACM, New York, pp 181–190

  • Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: ICSE ’05: proc. of the 27th international conference on software engineering. ACM, New York, pp 284–292

  • Nagappan N, Ball T (2007) Using software dependencies and churn metrics to predict field failures: an empirical case study. In: ESEM ’07: proceedings of the 1st international symposium on empirical software engineering and measurement. IEEE Computer Society, Washington, pp 364–373

  • Nagappan N, Ball T, Zeller A (2006) Mining metrics to predict component failures. In: ICSE ’06: proceedings of the 28th international conference on software engineering. ACM, New York, pp 452–461

  • Nguyen THD, Adams B, Hassan AE (2010) Studying the impact of dependency network measures on software quality. In: ICSM ’10: proceedings of the 2010 IEEE international conference on software maintenance. IEEE Computer Society, Washington, pp 1–10

  • Nguyen TT, Nguyen TN, Phuong TM (2011) Topic-based defect prediction (nier track). In: ICSE ’11: proceedings of the 33rd international conference on software engineering. ACM, New York, pp 932–935

  • Ohlsson N, Alberg H (1996) Predicting fault-prone software modules in telephone switches. IEEE Trans Softw Eng 22:886–894

    Article  Google Scholar 

  • Pinzger M, Nagappan N, Murphy B (2008) Can developer-module networks predict failures? In: SIGSOFT ’08/FSE-16: proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering. ACM, New York, pp 2–12

  • Posnett D, D’Souza R, Devanbu P, Filkov V (2013) Dual ecological measures of focus in software development. In: ICSE ’13: proceedings of the 2013 international conference on software engineering. IEEE Press, Piscataway, pp 452–461

  • Rabkin A, Katz R (2010) Chukwa: a system for reliable large-scale log collection. In: LISA’10: proc. of the 24th international conference on large installation system administration. USENIX, Berkeley, pp 1–15

  • Rahman F, Devanbu P (2011) Ownership, experience and defects: a fine-grained study of authorship. In: ICSE ’11: proceedings of the 33rd international conference on software engineering. ACM, New York, pp 491–500

  • Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: ICSE ’13: proceedings of the 2013 international conference on software engineering. IEEE Press, Piscataway, pp 432–441

  • Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008) Defect categorization: making use of a decade of widely varying historical data. In: ESEM ’08: proceedings of the 2nd ACM-IEEE international symposium on empirical software engineering and measurement. ACM, New York, pp 149–157

  • Shang W, Jiang ZM, Adams B, Hassan AE (2009) MapReduce as a general framework to support research in Mining Software Repositories (MSR). In: MSR ’09: proceedings of 6th IEEE international working conference on Mining Software Repositories, pp 21–30

  • Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2011) An exploratory study of the evolution of communicated information about the execution of large software systems. In: WCRE 2011: proceedings of the 2011 18th working conference on reverse engineering. IEEE Computer Society, Washington, pp 335–344

  • Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2011) An exploratory study of the evolution of communicated information about the execution of large software systems. In: WCRE ’11: proceedings of the 2011 18th working conference on reverse engineering. IEEE Computer Society, Washington, pp 335–344

  • Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2013) An exploratory study of the evolution of communicated information about the execution of large software systems. J Softw: Evolution and Process. doi:10.1109/WCRE.2011.48

  • Shihab E (2012) An exploration of challenges limiting pragmatic software defect prediction. Ph.D. thesis, Queen’s University

  • Shihab E, Jiang ZM, Ibrahim WM, Adams B, Hassan AE (2010) Understanding the impact of code and process metrics on post-release defects: a case study on the eclipse project. In: ESEM ’10: proc. of the 2010 ACM-IEEE int. symposium on empirical software engineering and measurement. ACM, New York, pp 4:1–4:10

  • Shihab E, Mockus A, Kamei Y, Adams B, Hassan AE (2011) High-impact defects: a study of breakage and surprise defects. In: ESEC/FSE ’11: proc. of the 19th ACM SIGSOFT symp. and the 13th Euro. conf. on foundations of software engineering. ACM, NY, pp 300–310

  • Smithson M (2003) Confidence intervals. Sage Publications, Thousand Oaks

    Book  Google Scholar 

  • Tan J, Pan X, Kavulya S, Gandhi R, Narasimhan P (2008) Salsa: analyzing logs as state machines. In: WASL’08: proceedings of the 1st USENIX conference on analysis of system logs. USENIX, San Diego, pp 6–6

  • Wolf T, Schroter A, Damian D, Nguyen T (2009) Predicting build failures using social network analysis on developer communication. In: ICSE ’09: proceedings of the 31st international conference on software engineering. IEEE Computer Society, Washington, pp 1–11

  • Xu W, Huang L, Fox A, Patterson D, Jordan MI (2009) Detecting large-scale system problems by mining console logs. In: SOSP ’09: proceedings of the ACM SIGOPS 22nd symposium on operating systems principles. ACM, Big Sky, Montana, pp 117–132

  • Yuan D, Park S, Zhou Y (2012) Characterizing logging practices in open-source software. In: ICSE 2012: proceedings of the 2012 international conference on software engineering, ICSE 2012. IEEE Press, Piscataway, pp 102–112

  • Yuan D, Zheng J, Park S, Zhou Y, Savage S (2011) Improving software diagnosability via log enhancement. In: ASPLOS ’11: proceedings of the 16th international conference on architectural support for programming languages and operating systems. ACM, Newport Beach, pp 3–14

  • Zhang F, Khomh F, Zou Y, Hassan AE (2012) An empirical study of the effect of file editing patterns on software quality. In: WCRE ’12: proceedings of the 2012 19th working conference on reverse engineering. IEEE Computer Society, Washington, pp 456–465

  • Zimmermann T, Nagappan N (2008) Predicting defects using network analysis on dependency graphs. In: ICSE ’08: proceedings of the 30th international conference on software engineering. ACM, New York, pp 531–540

  • Zimmermann T, Nagappan N, Williams L (2010) Searching for a needle in a haystack: Predicting security vulnerabilities for windows vista. In: Proceedings of the 2010 3rd international conference on software testing, verification and validation, ICST ’10, pp 421–428

  • Zimmermann T, Premraj R, Zeller A (2007) Predicting defects for eclipse. In: PROMISE ’07: proc. of the 3rd int. workshop on predictor models in software engineering. IEEE, Washington, pp 9–15

Download references

Author information

Authors and Affiliations

  1. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, ON, Canada

    Weiyi Shang, Meiyappan Nagappan & Ahmed E. Hassan

Authors
  1. Weiyi Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meiyappan Nagappan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiyi Shang.

Additional information

Communicated by: Sunghun Kim

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shang, W., Nagappan, M. & Hassan, A.E. Studying the relationship between logging characteristics and the code quality of platform software. Empir Software Eng 20, 1–27 (2015). https://doi.org/10.1007/s10664-013-9274-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-013-9274-8

Keywords

Search

Navigation