Abstract
Context
Reading and understanding the source code are fundamental to supporting software programmers’ daily activities. Still, there is no agreement regarding the program attributes needed to achieve the readability and comprehensibility of source code.
Objective
To analyze the influence of comments presence, indentation spacing, identifiers length, and code size on the readability and comprehensibility of source code from the perspective of novice and experienced programmers.
Method
We performed three primary studies and collected quantitative (Likert) and qualitative data representing the programmers’ judgment regarding the readability and comprehensibility of code snippets. For each study, the influence of the four attributes on the readability and comprehensibility of source code was analyzed. The results were aggregated using the odds-ratio approach and analyzed concerning participants’ programming experience.
Results
The quality characteristics were not significantly affected (alpha = 5%) by either the indentation spacing or the code size, whereas the presence of comments and identifier length affect source code quality positively under such characteristics, according to both novices and experienced programmers.
Conclusions
Although the results presented findings with statistical significance, the controlled factors and participants’ stratification between novices and experienced were insufficient to explain the contradictory findings in the technical literature concerning the impact of the attributes under evaluation on the readability and comprehensibility of source code.




Similar content being viewed by others
Data Availability
The datasets generated and analyzed during the current work are openly available in the Figshare repository at https://doi.org/10.6084/m9.figshare.22791104, reference number 22791104.
Notes
There are three source code attributes associated to each quality characteristic (see Section 3.1.4 for the list of quality characteristics evaluated for each attribute). Considering we are going to analyze results from three studies (observing that the analysis of S1 was split into two –novices and experienced), the 48 statistical tests come from 2 (quality characteristics) × 3 (source code attributes) × 4 (three studies with one split analysis) × 2 (perceptions and explanations/answers).
References
Alawad D, Panta M, Zibran M, Islam M (2019) "An empirical study of the relationships between code readability and software complexity," in arXiv preprint arXiv:1909.01760
Aman H (2012a) "An empirical analysis of the impact of comment statements on fault-proneness of small-size module," in Proceedings of the Asia-Pacific Software Engineering Conference, Hong Kong
Aman H (2012b) "An empirical analysis on fault-proneness of well-commented modules," in Proceedings of the International Workshop on Empirical Software Engineering in Practice, Osaka
Aman H, Amasaki S, Sasaki T, Kawahara M (2015) Lines of comments as a noteworthy metric for analyzing fault-proneness in method. IEICE Trans Inf Syst:2218–2228
Aman H, Amasaki S, Yokogawa T, Kawahara M (2016) "Local variables with compound names and comments as signs of fault-prone Java methods," in Proceedings of the International Workshop on Quantitative Approaches to Software Quality, Hamilton
Aman H, Amasaki S, Yokogawa T, Kawahara M (2017) "Empirical analysis of words in comments written for Java methods," in Proceedings of the Euromicro Conference on Software Engineering and Advanced Applications, Vienna
Avidan E, Feitelson DG (2017) "Effects of variable names on comprehension: an empirical study," in Proceedings of the International Conference on Program Comprehension, Buenos Aires
Basili VR, Caldiera G, Rombach HD (1994) Goal question metric paradigm, in Encyclopedia of Software Engineering. Wiley, New York, pp 528–532
Bauer J, Siegmund J, Peitek N, Hofmeister JC, Apel S (2019) "Indentation: simply a matter of style or support for program comprehension?," in Proceedings of the International Conference on Program Comprehension, Montreal
Beniamini G, Gingichashvili S, Orbach AK, Feitelson DG (2017) "Meaningful identifier names: the case of single-letter variables," in Proceedings of the International Conference on Program Comprehension, Buenos Aires
Binkley D, Feild H, Lawrie D, Pighin M (2007) "Software fault prediction using language processing," in Testing: Academic and Industrial Conference Practice and Research Techniques, Windsor
Binkley D, Feild H, Lawrie D, Pighin M (2009) Increasing diversity: natural language measures for software fault prediction. J Syst Softw 82:1793–1803
Borenstein M, Gedges LV, Higgins JPT, Rothstein HR (2009) Introduction to meta-analysis. Wiley, West Sussex
Börstler J, Paech B (2016) The role of method chains and comments in software readability and comprehension - An experiment. Transactions on Software Engineering 9:886–898
Bosch J (2014) "Continuous software engineering: an introduction," in Continuous Software Engineering, Springer, pp. 3–13
Buse RPL, Weimer WR (2008) "A metric for software readability," in Proceedings of the International Symposium on Software Testing and Analysis, Seattle
Buse RPL, Weimer WR (2010) Learning a metric for code readability. Transactions on Software Engineering 36:546–558
Butler S, Wermelinger M, Yu Y, Sharp H (2010) "Exploring the influence of identifier names on code quality: An empirical study," in Proceedings of European Conference on Software Maintenance and Reengineering, Madrid
Cohen J (1992) A power primer. Psychol Bull 112:155–159
DeYoung GE, Kampen GR (1979) "Program factors as predictors of program readability," in Proceedings of the International Computer Software and Applications Conference, Chicago
Eaddy M, Zimmermann T, Sherwood KD, Garg V, Murphy GC, Nagappan N, Aho AV (2008) Do crosscutting concerns cause defects? Transactions on Software Engineering 34:497–515
Feigenspan J, Apel S, Liebig J, Kästner C (2011) "Exploring software measures to assess program comprehension," in Proceedings of the International Symposium on Empirical Software Engineering and Measurement, Banff
Garvin D (1984) What does “product quality” really mean? Sloan Manag Rev 25:25–43
Glass RL (2002) Facts and fallacies of software engineering, 1 ed. Addison-Wesley, Boston, Massachusetts, p 224
Guerrouj L, Kermansaravi Z, Arnaoudova V, Fung B, Khomh F, Antoniol G, Guéhéneuc Y-G (2017) Investigating the relation between lexical smells and change and fault-proneness: an empirical study. Softw Qual J 25:641–670
Hofmeister J, Siegmund J, Holt DV (2017) Shorter identifier names take longer to comprehend, in International Conference on Software Analysis. Evolution and Reengineering, Klagenfurt
Hofmeister JC, Siegmund J, Holt DV (2019) Shorter identifier names take longer to comprehend. Empir Softw Eng 24:417–443
Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering, in Guide to advanced empirical software engineering. Springer, London, pp 201–228
Jørgensen AH (1980) A methodology for measuring the readability and modifiability of computer programs. BIT Numer Math 20:393–405
Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49:1073–1086
Lawrie D, Morrell C, Feild H, Binkley D (2006) "What's in a name? A study of identifiers," in Proceedings of the International Conference on Program Comprehension, Athens
Lawrie D, Morrell C, Feild H, Binkley D (2007) Effective identifier names for comprehension and memory. Innov Syst Softw Eng 3:303–318
Lee T, Lee J, In H (2015) Effect analysis of coding convention violations on readability of post-delivered code. IEICE Trans Inf Syst E98-D:1286–1296
Lewis S, Clarke M (2001) Forest plots: trying to see the wood and the trees. BMJ 322:1479–1480
Lientz BP, Swanson EB, Tompkins AGE (1978) Characteristics of application software maintenance. Communications of ACM 21:466–471
Lipsey MW, Wilson DB (2001) Practical meta-analysis - applied social research methods series, vol. 49. Sage Publications, London
Martínez-Fernández S, Santos PSMD, Ayala CP, Franch X, Travassos GH (2015) "Aggregating empirical evidence about the benefits and drawbacks of software reference architectures," in International Symposium on Empirical Software Engineering, Beijing
Miara RJ, Musselman JA, Navarro JA, Shneiderman B (1983) Program indentation and comprehensibility. Commun ACM 26:861–867
Miyake Y, Amasaki S, Aman H, Yokogawa T (2017) "A replicated study on relationship between code quality and method comments," Applied Computing and Information Technology Studies in Computational Intelligence, pp. 17–30
Nguyen V, Boehm B, Danphitsanuphan P (2011) A controlled experiment in assessing and estimating software maintenance tasks. Inf Softw Technol 53:682–691
Nielebock S, Krolikowski D, Krüger J, Leich T, Ortmeier F (2019) Commenting source code: is it worth it for small programming tasks? Empir Softw Eng 24:1418–1457
Olivier J, Bell ML (2013) Effect sizes for 2×2 contingency tables. PLoS One 8
Peitek N, Apel S, Parnin C, Brechmann A, Siegmund J (2021) "Program comprehension and code complexity metrics: an fMRI study," in International Conference on Software Engineering, Madrid
Petersen K (2010) "Implementing lean and agile software development in industry," Karlskrona
Posnett D, Hindle A, Devanbu P (2011) "A simpler model of software readability," in Proceedings of the Working Conference on Mining Software Repositories, Waikiki
Raymond DR (1991) "Reading source code," in Proceedings of the Conference of the Centre for Advanced Studies on Collaborative Research, Toronto
Ribeiro TV, Travassos GH (2015) "On the alignment of source code quality perspectives through experimentation: an industrial case," in Proceedings of the International Workshop on Conducting Empirical Studies in Industry (CESI/ICSE 2015), Florence
Ribeiro TV, Travassos GH (2017) "Who is right? Evaluating empirical contradictions in the readability and comprehensibility of source code," in XX Ibero-American Conference on Software Engineering, Buenos Aires
Ribeiro TV, Travassos GH (2018) Attributes influencing the reading and comprehension of source code – discussing contradictory evidence. CLEI Electron J 21
Rugaber S (2000) The use of domain knowledge in program understanding. Ann Softw Eng 9:143–192
Salviulo F, Scanniello G (2014) Dealing with identifiers and comments in source code comprehension and maintenance: Results from an ethnographically-informed study with students and professionals. EASE '14: proceedings of the 18th international conference on evaluation and assessment in software engineering, p 10. https://doi.org/10.1145/2601248.2601251
Santos RMD, Gerosa MA (2018) "Impacts of coding practices on readability," in Proceedings of the International Conference on Program Comprehension, Gothenburg
Scalabrino S, Bavota G, Vendome C, Linares-Vásquez M, Poshyvanyk D, Oliveto R (2017) "Automatically assessing code understandability: How far are we?," in International Conference on Automated Software Engineering, Urbana
Scalabrino S, Bavota G, Vendome C, Linares-Vásquez M, Poshyvanyk D, Oliveto R (2021) Automatically assessing code understandability. Transactions on Software Engineering 47:595–613
Scalabrino S, Linares-Vásquez M, Oliveto R, Poshyvanyk D (2018) A comprehensive model for code readability. J Softw Evol Process 30:1–23
Scanniello G, Risi M (2013) "Dealing with faults in source code: abbreviated vs. full-word identifier names," in Proceedings of the International Conference on Software Maintenance, Eindhoven
Scanniello G, Risi M, Tramontana P, Romano S (2017) Fixing faults in C and Java source code: Abbreviated vs. full-word identifier names. ACM Trans Softw Eng Methodol 26:1–43
Siegmund J, Kästner C, Jörg Liebig SA, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19:1299–1334
Tashtoush Y, Odat Z, Alsmadi I, Yatim M (2013) Impact of programming features on code readability. Int J Softw Eng Appl 7:441–458
Tenny T (1988) Program readability: procedures versus comments. Transactions on Software Engineering 14:1271–1279
Trockman A, Cates K, Cates K, Nguyen T, Kästner C, Vasilescu B (2018) ""Automatically assessing code understandability" reanalyzed: combined metrics matter," in Proceedings of the International Conference on Mining Software Repositories, Gothenburg
Woodfield S, Dunsmore H, Shen V (1981) "Effect of modularization and comments on program comprehension," in Proceedings of the International Conference on Software Engineering, San Diego
Xia X, Bao L, Xing Z, Hassan AE, Li S (2018) Measuring program comprehension: A large-scale field study with professionals. Transactions on Software Engineering 44:951–976
Funding
The authors thank CAPES and CNPq for supporting this research. Prof. Travassos is a CNPq Researcher and CNE FAPERJ.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors thank the students of the courses in which we conducted the experiments (EEL873 – S1, CPS820 – S2, MAB114 – S3, and MAB121 – S3) at UFRJ for taking part in the three different studies.
The authors declare they have no financial interests and no competing interests to declare that are relevant to the content of this article.
Additional information
This article belongs to the Topical Collection: Special Issue on Code Legibility, Readability, and Understandability.
Appendix A
Appendix A
References for the works mentioned in the Background section.
A | DeYoung and Kampen (1979) |
B | Jørgensen (1980) |
C | Woodfield et al. (1981) |
D | Miara et al. (1983) |
E | Tenny (1988) |
F | Lawrie et al. (2006) |
G | Lawrie et al. (2007) |
H | Buse and Weimer (2008) |
I | Buse and Weimer (2010) |
J | Butler et al. (2010) |
K | Feigenspan et al. (2011) |
L | Posnett et al. (2011) |
M | Tashtoush et al. (2013) |
N | Lee et al. (2015) |
O | Börstler and Paech (2016) |
a | Avidan and Feitelson (2017) |
b | Beniamini et al. (2017) |
c | Scalabrino et al. (2017) |
d | Hofmeister et al. (2017) |
e | Scalabrino et al. (2018) |
f | Trockman et al. (2018) |
g | Nielebock et al. (2019) |
h | Bauer et al. (2019) |
i | Hofmeister et al. (2019) |
j | Scalabrino et al. (2021) |
k | Peitek et al. (2021) |
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ribeiro, T.V., dos Santos, P.S.M. & Travassos, G.H. On the Investigation of Empirical Contradictions - Aggregated Results of Local Studies on Readability and Comprehensibility of Source Code. Empir Software Eng 28, 148 (2023). https://doi.org/10.1007/s10664-023-10360-5
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-023-10360-5