Skip to main content
SpringerLink
Log in

Measuring code maintainability with deep neural networks

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

The maintainability of source code is a key quality characteristic for software quality. Many approaches have been proposed to quantitatively measure code maintainability. Such approaches rely heavily on code metrics, e.g., the number of Lines of Code and McCabe’s Cyclomatic Complexity. The employed code metrics are essentially statistics regarding code elements, e.g., the numbers of tokens, lines, references, and branch statements. However, natural language in source code, especially identifiers, is rarely exploited by such approaches. As a result, replacing meaningful identifiers with nonsense tokens would not significantly influence their outputs, although the replacement should have significantly reduced code maintainability. To this end, in this paper, we propose a novel approach (called DeepM) to measure code maintainability by exploiting the lexical semantics of text in source code. DeepM leverages deep learning techniques (e.g., LSTM and attention mechanism) to exploit these lexical semantics in measuring code maintainability. Another key rationale of DeepM is that measuring code maintainability is complex and often far beyond the capabilities of statistics or simple heuristics. Consequently, DeepM leverages deep learning techniques to automatically select useful features from complex and lengthy inputs and to construct a complex mapping (rather than simple heuristics) from the input to the output (code maintainability index). DeepM is evaluated on a manually-assessed dataset. The evaluation results suggest that DeepM is accurate, and it generates the same rankings of code maintainability as those of experienced programmers on 87.5% of manually ranked pairs of Java classes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Schneidewind N F. The state of software maintenance. IEEE Transactions on Software Engineering, 1987, SE-13(3): 303–310

    Article  MATH  Google Scholar 

  2. Bennett K H, Rajlich V T. Software maintenance and evolution: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering. 2000, 73–87

  3. Lientz B P, Swanson E B. Software Maintenance Management: A Study of the Maintenance of Computer Application Software in 487 Data Processing Organizations. Reading: Addison-Wesley, 1980

    Google Scholar 

  4. Yau S S, Collofello J S. Some stability measures for software maintenance. IEEE Transactions on Software Engineering, 1980, SE-6(6): 545–552

    Article  Google Scholar 

  5. Reiss S P. Incremental maintenance of software artifacts. IEEE Transactions on Software Engineering, 2006, 32(9): 682–697

    Article  Google Scholar 

  6. ISO. ISO/IEC 5055: 2021 Information technology — Software measurement — Software quality measurement — Automated source code quality measures. Geneva: ISO, 2021

  7. Grubb P, Takang A A. Software Maintenance: Concepts and Practice. 2nd ed. London: World Scientific Publishing, 2003

    Book  MATH  Google Scholar 

  8. Al Dallal J. Object-oriented class maintainability prediction using internal quality attributes. Information and Software Technology, 2013, 55(11): 2028–2048

    Article  Google Scholar 

  9. Myers G J, Badgett T, Thomas T M, Sandler C. The Art of Software Testing. 2nd ed. Hoboken: John Wiley & Sons, 2004

    Google Scholar 

  10. Mari M, Eila N. The impact of maintainability on component-based software systems. In: Proceedings of the 29th Euromicro Conference. 2003, 25–32

  11. Li W, Henry S. Object-oriented metrics that predict maintainability. Journal of Systems and Software, 1993, 23(2): 111–122

    Article  Google Scholar 

  12. Oman P, Hagemeister J. Construction and testing of polynomials predicting software maintainability. Journal of Systems and Software, 1994, 24(3): 251–266

    Article  Google Scholar 

  13. Heitlager I, Kuipers T, Visser J. A practical model for measuring maintainability. In: Proceedings of the 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007). 2007, 30–39

  14. ISO. ISO/IEC 9126-1: 2001 Software engineering—Product quality—Part 1: Quality model. Geneva: ISO, 2001

    Google Scholar 

  15. Zhou Y, Leung H. Predicting object-oriented software maintainability using multivariate adaptive regression splines. Journal of Systems and Software, 2007, 80(8): 1349–1361

    Article  Google Scholar 

  16. Bavota G, Dit B, Oliveto R, Di Penta M, Poshyvanyk D, De Lucia A. An empirical study on the developers’ perception of software coupling. In: Proceedings of the 35th International Conference on Software Engineering (ICSE). 2013, 692–701

  17. Pantiuchina J, Lanza M, Bavota G. Improving code: the (Mis) perception of quality metrics. In: Proceedings of 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). 2018, 80–91

  18. Lenhard J, Blom M, Herold S. Exploring the suitability of source code metrics for indicating architectural inconsistencies. Software Quality Journal, 2019, 27(1): 241–274

    Article  Google Scholar 

  19. Liu H, Xu Z, Zou Y. Deep learning based feature envy detection. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 385–396

  20. Scalabrino S, Linares-Vasquez M, Poshyvanyk D, Oliveto R. Improving code readability models with textual features. In: Proceedings of the 24th IEEE International Conference on Program Comprehension. 2016, 1–10

  21. Kim S, Kim D. Automatic identifier inconsistency detection using code dictionary. Empirical Software Engineering, 2016, 21(2): 565–604

    Article  Google Scholar 

  22. Schankin A, Berger A, Holt D V, Hofmeister J C, Riedel T, Beigl M. Descriptive compound identifier names improve source code comprehension. In: Proceedings of the 26th IEEE/ACM International Conference on Program Comprehension (ICPC). 2018, 31–40

  23. Lai S, Liu K, He S, Zhao J. How to generate a good word embedding. IEEE Intelligent Systems, 2016, 31(6): 5–14

    Article  Google Scholar 

  24. Greff K, Srivastava R K, Koutník J, Steunebrink B R, Schmidhuber J. LSTM: a search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(10): 2222–2232

    Article  Google Scholar 

  25. Tai K S, Socher R, Manning C D. Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015, 1556–1566

  26. Raffel C, Ellis D P. Feed-forward networks with attention can solve some long-term memory problems. 2016, arXiv preprint arXiv: 1512.08756

  27. McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 1943, 5(4): 115–133

    Article  MATH  Google Scholar 

  28. Nuñez-Varela A S, Pérez-Gonzalez H G, Martínez-Perez F E, Soubervielle-Montalvo C. Source code metrics: a systematic mapping study. Journal of Systems and Software, 2017, 128: 164–197

    Article  Google Scholar 

  29. Przybyłek A. Where the truth lies: AOP and its impact on software modularity. In: Proceedings of the 14th International Conference on Fundamental Approaches to Software Engineering. 2011, 447–461

  30. Goel B M, Bhatia P K. Analysis of reusability of object-oriented systems using object-oriented metrics. ACM SIGSOFT Software Engineering Notes, 2013, 38(4): 1–5

    Article  Google Scholar 

  31. Bruntink M, Van Deursen A. An empirical study into class testability. Journal of Systems and Software, 2006, 79(9): 1219–1232

    Article  Google Scholar 

  32. Poort E R, Martens N, Van De Weerd I, Van Vliet H. How architects see non-functional requirements: beware of modifiability. In: Proceedings of the 18th International Conference on Requirements Engineering: Foundation for Software Quality. 2012, 37–51

  33. Alzahrani M, Alqithami S, Melton A. Using client-based class cohesion metrics to predict class maintainability. In: Proceedings of the 43rd IEEE Annual Computer Software and Applications Conference (COMPSAC). 2019, 72–80

  34. Kanellopoulos Y, Antonellis P, Antoniou D, Makris C, Theodoridis E, Tjortjis C, Tsirakis N. Code quality evaluation methodology using the ISO/IEC 9126 standard. International Journal of Software Engineering & Applications, 2010, 1(3): 17–36

    Article  Google Scholar 

  35. Malhotra R, Lata K. An empirical study to investigate the impact of data resampling techniques on the performance of class maintainability prediction models. Neurocomputing, 2021, 459: 432–453

    Article  Google Scholar 

  36. Padhy N, Panigrahi R, Neeraja K. Threshold estimation from software metrics by using evolutionary techniques and its proposed algorithms, models. Evolutionary Intelligence, 2019, 14(2): 315–329

    Article  Google Scholar 

  37. Shatnawi R. Comparison of threshold identification techniques for object-oriented software metrics. IET Software, 2020, 14(6): 727–738

    Article  Google Scholar 

  38. Ferreira K A M, Bigonha M A S, Bigonha R S, Mendes L F O, Almeida H C. Identifying thresholds for object-oriented software metrics. Journal of Systems and Software, 2012, 85(2): 244–257

    Article  Google Scholar 

  39. Hofmeister J, Siegmund J, Holt D V. Shorter identifier names take longer to comprehend. In: Proceedings of the 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2017, 217–227

  40. Morales R, Khomh F, Antoniol G. RePOR: mimicking humans on refactoring tasks. Are we there yet? Empirical Software Engineering, 2020, 25(4): 2960–2996

    Article  Google Scholar 

  41. Hussain Y, Huang Z, Zhou Y. Improving source code suggestion with code embedding and enhanced convolutional long short-term memory. IET Software, 2021, 15(3): 199–213

    Article  Google Scholar 

  42. Scalabrino S, Bavota G, Vendome C, Linares-Vasquez M, Poshyvanyk D, Oliveto R. Automatically assessing code understandability. IEEE Transactions on Software Engineering, 2021, 47(3): 595–613

    Article  Google Scholar 

  43. Smith N, Van Bruggen D, Tomassetti F. JavaParser: Visited. Victoria: Leanpub, 2021

    Google Scholar 

  44. Jiang Y, Liu H, Zhu J, Zhang L. Automatic and accurate expansion of abbreviations in parameters. IEEE Transactions on Software Engineering, 2020, 46(7): 732–747

    Article  Google Scholar 

  45. Butler S. The effect of identifier naming on source code readability and quality. In: Proceedings of the Doctoral Symposium for ESEC/FSE on Doctoral Symposium. 2009, 33–34

  46. Lawrie D, Feild H, Binkley D. Quantifying identifier quality: an analysis of trends. Empirical Software Engineering, 2007, 12(4): 359–388

    Article  Google Scholar 

  47. ORACLE. Naming Conventions. See oracle-base/articles/misc/naming-conventions website, 2017

  48. Olsson M. Constants. In: Olsson M, ed. Java 17 Quick Syntax Reference. 3rd ed. Berkele: Apress Berkele, 2022, 85–87

    Chapter  Google Scholar 

  49. Fowler M. Refactoring: Improving the Design of Existing Code. 2nd ed. Reading: Addison-Wesley Professional, 2018

    MATH  Google Scholar 

  50. Mi Q, Xiao Y, Cai Z, Jia X. The effectiveness of data augmentation in code readability classification. Information and Software Technology, 2021, 129: 106378

    Article  Google Scholar 

  51. Alon U, Zilberstein M, Levy O, Yahav E. Code2vec: learning distributed representations of code. Proceedings of the ACM on Programming Languages, 2019, 3(POPL): 40

    Article  Google Scholar 

  52. Lee T, Lee J B, In H P. A study of different coding styles affecting code readability. International Journal of Software Engineering and its Applications, 2013, 7(5): 413–422

    Article  Google Scholar 

  53. Sun H, Wang R, Chen K, Utiyama M, Sumita E, Zhao T. Unsupervised bilingual word embedding agreement for unsupervised neural machine translation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 1235–1245

  54. Chidamber S R, Kemerer C F. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 1994, 20(6): 476–493

    Article  Google Scholar 

  55. Aniche M. Java code metrics calculator (CK), 2015

  56. Svyatkovskiy A, Deng S K, Fu S, Sundaresan N. IntelliCode compose: code generation using transformer. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2020, 1433–1443

  57. Svyatkovskiy A, Zhao Y, Fu S, Sundaresan N. Pythia: AI-assisted code completion system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019, 2727–2735

  58. Li Y, Yang Z, Guo Y, Chen X. Humanoid: a deep learning-based approach to automated black-box android app testing. In: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 2019, 1070–1073

  59. Gu X, Zhang H, Zhang D, Kim S. Deep API learning. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2016, 631–642

  60. Goodfellow I, Bengio Y, Courville A. Deep Learning. Cambridge: MIT Press, 2016

    MATH  Google Scholar 

  61. Lever J, Krzywinski M, Altman N. Logistic regression. Nature Methods, 2016, 13(7): 541–543

    Article  Google Scholar 

  62. Zhang Y, Zhou M, Mockus A, Jin Z. Companies’ participation in OSS development-an empirical study of OpenStack. IEEE Transactions on Software Engineering, 2021, 47(10): 2242–2259

    Article  Google Scholar 

  63. Amreen S, Mockus A, Zaretzki R, Bogart C, Zhang Y X. ALFAA: active learning fingerprint based anti-aliasing for correcting developer identity errors in version control systems. Empirical Software Engineering, 2020, 25(2): 1136–1167

    Article  Google Scholar 

  64. El Emam K, Benlarbi S, Goel N, Rai S N. The confounding effect of class size on the validity of object-oriented metrics. IEEE Transactions on Software Engineering, 2001, 27(7): 630–650

    Article  Google Scholar 

  65. Bengio Y. Practical recommendations for gradient-based training of deep architectures. In: Montavon G, Orr G B, Müller K R, eds. Neural Networks: Tricks of the Trade. Berlin: Springer, 2012, 437–478

    Chapter  Google Scholar 

  66. Wang W, Li G, Shen S, Xia X, Jin Z. Modular tree network for source code representation learning. ACM Transactions on Software Engineering and Methodology, 2020, 29(4): 31

    Article  Google Scholar 

  67. Allamanis M, Barr E T, Bird C, Sutton C. Suggesting accurate method and class names. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. 2015, 38–49

  68. Baranasuriya N. Java Coding Standard. See se-education/guides/conventions/java/index website, 2022

  69. Triguero I, González S, Moyano J M, García S, Alcalá-Fdez J, Luengo J, Fernández A, Del Jesús M J, Sánchez L, Herrera F. KEEL 3.0: an open source software for multi-stage analysis in data mining. International Journal of Computational Intelligence Systems, 2017, 10(1): 1238–1249

    Article  Google Scholar 

  70. Zerouali A, Mens T. Analyzing the evolution of testing library usage in open source java projects. In: Proceedings of the 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2017, 417–421

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Beijing Institute of Technology, Beijing, 100081, China

    Yamin Hu & Zongyao Hu

  2. School of Artificial Intelligence, Anhui University, Hefei, 230601, China

    Hao Jiang

Authors
  1. Yamin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zongyao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yamin Hu.

Additional information

Yamin Hu received a BSc degree from the College of Information Engineering, Northwest A&F University, China in 2016, and MSc degree from the School of Computer Science and Technology, University of Science and Technology of China, China in 2019. He is currently working toward a PhD degree at the School of Computer Science and Technology, Beijing Institute of Technology, China. His current research interests include software engineering and database systems.

Hao Jiang received a BSc degree from the Department of Computer Science and Technology, North China Electric Power University, China in 2014, and PhD degree from the School of Computer Science and Technology, University of Science and Technology of China, China. He is currently an Associate Professor with the School of Artificial Intelligence, Anhui University, China. His current research interests include complex network analysis, combinatorial optimization, and computational intelligence.

Zongyao Hu is currently pursuing the PhD degree with the School of Computer Science and Technology, Beijing Institute of Technology, China. His current research interests include deep learning and image processing.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, Y., Jiang, H. & Hu, Z. Measuring code maintainability with deep neural networks. Front. Comput. Sci. 17, 176214 (2023). https://doi.org/10.1007/s11704-022-2313-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-022-2313-0

Keywords

Search

Navigation