Skip to main content
SpringerLink
Log in

Code Reviews, Software Inspections, and Code Walkthroughs: Systematic Mapping Study of Research Topics

  • Conference paper
  • First Online:
Software Quality: Quality Intelligence in Software and Systems Engineering (SWQD 2020)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 371))

Included in the following conference series:

Abstract

Code reviews have been used to improve code quality since the 1970s. Most practitioners in the field of software have some experience with respect to the technique. In this mapping study we illustrate what kinds of research questions are addressed in code review literature. The following themes emerged from analysis of 75 original articles: (1) description or comparison of different code review practices, (2) behavior of reviewers (e.g., eye tracking studies), (3) communication and teamwork, (4) outcomes of code reviews (e.g., what kinds of problems are identified), (5) how properties of code to be reviewed affect reviewing, and (6) reasons for conducting code reviews. About half of the studies have been conducted with students and novices. The numbers of industry papers has significantly increased when compared to the previous reviews in the field.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Myers, G.J.: A controlled experiment in program testing and code walkthroughs/inspections. Commun. ACM 21(9), 760–768 (1978)

    Article  Google Scholar 

  2. Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 International Conference on Software engineering, pp. 712–721. IEEE Press (2013)

    Google Scholar 

  3. Brykczynski, B.: A survey of software inspection checklists. ACM SIGSOFT Softw. Eng. Notes 24(1), 82 (1999)

    Article  Google Scholar 

  4. Ebad, S.: Inspection reading techniques applied to software artifacts-a systematic review. Comput. Syst. Sci. Eng. 32(3), 213–226 (2017)

    Google Scholar 

  5. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: Proc. of the 12th International Conference on Evaluation and Assessment in Software Engineering. EASE 2008, BCS Learning & Development, pp. 68–77 (2008)

    Google Scholar 

  6. Kitchenham, B.A., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE-2007-01, Keele Univ. (2007)

    Google Scholar 

  7. Greiler, M., Bird, C., Storey, M.A., MacLeod, L., Czerwonka, J.: Code reviewing in the trenches: Understanding challenges, best practices and tool needs (2016)

    Google Scholar 

  8. Dunsmore, A., Roper, M., Wood, M.: Systematic object-oriented inspection-an empirical study. In: Proceedings of the 23rd International Conference on Software Engineering, pp. 135–144. IEEE (2001)

    Google Scholar 

  9. Edmundson, A., Holtkamp, B., Rivera, E., Finifter, M., Mettler, A., Wagner, D.: An empirical study on the effectiveness of security code review. In: Jürjens, J., Livshits, B., Scandariato, R. (eds.) ESSoS 2013. LNCS, vol. 7781, pp. 197–212. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36563-8_14

    Chapter  Google Scholar 

  10. Begel, A., Vrzakova, H.: Eye movements in code review. In: Proceedings of the Workshop on Eye Movements in Programming, p. 5. ACM (2018)

    Google Scholar 

  11. Uwano, H., Nakamura, M., Monden, A., Matsumoto, K.I.: Analyzing individual performance of source code review using reviewers’ eye movement. In: Proceedings of the 2006 Symposium on Eye Tracking Research & Applications, pp. 133–140. ACM (2006)

    Google Scholar 

  12. Foss, S.L., Murphy, G.C.: Do developers respond to code stability warnings? In: Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering, pp. 162–170. IBM Corp. (2015)

    Google Scholar 

  13. dos Santos, E.W., Nunes, I.: Investigating the effectiveness of peer code review in distributed software development. In: Proceedings of the 31st Brazilian Symposium on Software Engineering, pp. 84–93. ACM (2017)

    Google Scholar 

  14. Spadini, D., Aniche, M., Storey, M.A., Bruntink, M., Bacchelli, A.: When testing meets code review: why and how developers review tests. In: 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pp. 677–687. IEEE (2018)

    Google Scholar 

  15. Sripada, S.K., Reddy, Y.R.: Code comprehension activities in undergraduate software engineering course-a case study. In: 24th Australasian Software Engineering Conference, vol. 2015, pp. 68–77. IEEE (2015)

    Google Scholar 

  16. Baysal, O., Kononenko, O., Holmes, R., Godfrey, M.W.: The influence of non-technical factors on code review. In: 2013 20th Working Conference on Reverse Engineering (WCRE), pp. 122–131. IEEE (2013)

    Google Scholar 

  17. Sadowski, C., Söderberg, E., Church, L., Sipko, M., Bacchelli, A.: Modern code review: a case study at google. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, pp. 181–190. ACM (2018)

    Google Scholar 

  18. Fracz, W., Dajda, J.: Experimental validation of source code reviews on mobile devices. In: Gervasi, O., Murgante, B., Misra, S., Borruso, G., Torre, C.M., Rocha, A.M.A.C., Taniar, D., Apduhan, B.O., Stankova, E., Cuzzocrea, A. (eds.) ICCSA 2017. LNCS, vol. 10408, pp. 533–547. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-62404-4_40

    Chapter  Google Scholar 

  19. Baum, T., Leßmann, H., Schneider, K.: The choice of code review process: a survey on the state of the practice. In: Felderer, M., Méndez Fernández, D., Turhan, B., Kalinowski, M., Sarro, F., Winkler, D. (eds.) PROFES 2017. LNCS, vol. 10611, pp. 111–127. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69926-4_9

    Chapter  Google Scholar 

  20. Ferreira, A.L., Machado, R.J., Silva, J.G., Batista, R.F., Costa, L., Paulk, M.C.: An approach to improving software inspections performance. In: 2010 IEEE International Conference on Software Maintenance, pp. 1–8. IEEE (2010)

    Google Scholar 

  21. Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Zaidman, A., Gall, H.C.: Context is king: the developer perspective on the usage of static analysis tools. In: IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 38–49. IEEE (2018)

    Google Scholar 

  22. Höst, M., Johansson, C.: Evaluation of code review methods through interviews and experimentation. J. Syst. Softw. 52(2–3), 113–120 (2000)

    Article  Google Scholar 

  23. Kamsties, E., Lott, C.M.: An empirical evaluation of three defect-detection techniques. In: Schäfer, W., Botella, P. (eds.) Software Engineering – ESEC 1995, pp. 362–383. Springer, Berlin (1995)

    Chapter  Google Scholar 

  24. Müller, M.M.: Are reviews an alternative to pair programming? Empir. Softw. Eng. 9(4), 335–351 (2004)

    Article  Google Scholar 

  25. Khandelwal, S., Sripada, S.K., Reddy, Y.R.: Impact of gamification on code review process: an experimental study. In: Proceedings of the 10th Innovations in Software Engineering Conference, pp. 122–126. ACM (2017)

    Google Scholar 

  26. Hatton, L.: Testing the value of checklists in code inspections. IEEE Softw. 25(4), 82–88 (2008)

    Article  Google Scholar 

  27. Belli, F., Crisan, R.: Empirical performance analysis of computer-supported code-reviews. In: Proceedings The Eighth International Symposium on Software Reliability Engineering, pp. 245–255. IEEE (1997)

    Google Scholar 

  28. El Emam, K., Laitenberger, O.: Evaluating capture-recapture models with two inspectors. IEEE Trans. Softw. Eng. 27(9), 851–864 (2001)

    Article  Google Scholar 

  29. Hirao, T., Ihara, A., Matsumoto, K.I.: Pilot study of collective decision-making in the code review process. In: Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering, pp. 248–251. IBM Corp (2015)

    Google Scholar 

  30. Olorisade, B.K., Vegas, S., Juristo, N.: Determining the effectiveness of three software evaluation techniques through informal aggregation. Inf. Softw. Technol. 55(9), 1590–1601 (2013)

    Article  Google Scholar 

  31. Runeson, P., Stefik, A., Andrews, A., Gronblom, S., Porres, I., Siebert, S.: A comparative analysis of three replicated experiments comparing inspection and unit testing. In: Second International Workshop on Replication in Empirical Software Engineering Research, vol. 2011, pp. 35–42. IEEE (2011)

    Google Scholar 

  32. De Vreede, G.J., Koneri, P.G., Dean, D.L., Fruhling, A.L., Wolcott, P.: A collaborative software code inspection: the design and evaluation of a repeatable collaboration process in the field. Int. J. Coop. Inf. Syst. 15(02), 205–228 (2006)

    Article  Google Scholar 

  33. Hémeury, B.: Report on the VERA experiment. In: González Harbour, M., de la Puente, J.A. (eds.) Ada-Europe 1999. LNCS, vol. 1622, pp. 103–113. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-48753-0_9

    Chapter  Google Scholar 

  34. Kelly, D., Shepard, T.: Qualitative observations from software code inspection experiments. In: Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research, p. 5. IBM Press (2002)

    Google Scholar 

  35. Porter, A.A., Siy, H.P., Toman, C.A., Votta, L.G.: An experiment to assess the cost-benefits of code inspections in large scale software development. IEEE Trans. Softw. Eng. 23(6), 329–346 (1997)

    Article  Google Scholar 

  36. Wang, Y.Q., Qi, Z.Y., Zhang, L.J., Song, M.J.: Research and practice on education of SQA at source code level. Int. J. Eng. Educ. 27(1), 70 (2011)

    Google Scholar 

  37. Panko, R.R.: Applying code inspection to spreadsheet testing. J. Manage. Inf. Syst. 16(2), 159–176 (1999)

    Article  Google Scholar 

  38. Koneri, P.G., de Vreede, G.-J., Dean, D.L., Fruhling, A.L., Wolcott, P.: The design and field evaluation of a repeatable collaborative software code inspection process. In: Fukś, H., Lukosch, S., Salgado, A.C. (eds.) CRIWG 2005. LNCS, vol. 3706, pp. 325–340. Springer, Heidelberg (2005). https://doi.org/10.1007/11560296_26

    Chapter  Google Scholar 

  39. Cristia, M., Frydman, C.: Formal and semi-formal verification of a web voting system. Int. J. Web Inf. Syst. 11(2), 183–204 (2015)

    Article  Google Scholar 

  40. da Silva Neto, A.V., Vismari, L.F., Gimenes, R.A.V., Sesso, D.B., de Almeida, J.R., Cugnasca, P.S., Camargo, J.B.: A practical analytical approach to increase confidence in pld-based systems safety analysis. IEEE Syst. J. 99, 1–12 (2017)

    Google Scholar 

  41. Wood, M., Roper, M., Brooks, A., Miller, J.: Comparing and combining software defect detection techniques: a replicated empirical study. In: Jazayeri, M., Schauer, H. (eds.) ESEC/SIGSOFT FSE -1997. LNCS, vol. 1301, pp. 262–277. Springer, Heidelberg (1997). https://doi.org/10.1007/3-540-63531-9_19

    Chapter  Google Scholar 

  42. Wilkerson, J.W., Nunamaker, J.F., Mercer, R.: Comparing the defect reduction benefits of code inspection and test-driven development. IEEE Trans. Softw. Eng. 38(3), 547–560 (2011)

    Article  Google Scholar 

  43. Morales, R., McIntosh, S., Khomh, F.: Do code review practices impact design quality? a case study of the qt, vtk, and itk projects. In: IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 171–180 IEEE (2015)

    Google Scholar 

  44. Oliveira, R., Estácio, B., Garcia, A., Marczak, S., Prikladnicki, R., Kalinowski, M., Lucena, C.: Identifying code smells with collaborative practices: a controlled experiment. In: X Brazilian Symposium on Software Components, Architectures and Reuse (SBCARS), pp. 61–70. IEEE 2016 (2016)

    Google Scholar 

  45. Swamidurai, R., Dennis, B., Kannan, U.: Investigating the impact of peer code review and pair programming on test-driven development. In: IEEE SOUTHEASTCON 2014, pp. 1–5. IEEE (2014)

    Google Scholar 

  46. Bisant, D.B., Lyle, J.R.: A two-person inspection method to improve programming productivity. IEEE Trans. Softw. Eng. 10, 1294–1304 (1989)

    Article  Google Scholar 

  47. McMeekin, D.A., von Konsky, B.R., Chang, E., Cooper, D.J.A.: Measuring cognition levels in collaborative processes for software engineering code inspections. In: Ulieru, M., Palensky, P., Doursat, R. (eds.) IT Revolutions 2008. LNICST, vol. 11, pp. 32–43. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03978-2_5

    Chapter  Google Scholar 

  48. McMeekin, D.A., von Konsky, B.R., Chang, E., Cooper, D.J.: Checklist based reading’s influence on a developer’s understanding. In: 19th Australian Conference on Software Engineering (aswec 2008), pp. 489–496. IEEE (2008)

    Google Scholar 

  49. McIntosh, S., Kamei, Y., Adams, B., Hassan, A.E.: An empirical study of the impact of modern code review practices on software quality. Empir. Softw. Eng. 21(5), 2146–2189 (2016)

    Article  Google Scholar 

  50. Stålhane, T., Awan, T.H.: Improving the software inspection process. In: Richardson, I., Abrahamsson, P., Messnarz, R. (eds.) EuroSPI 2005. LNCS, vol. 3792, pp. 163–174. Springer, Heidelberg (2005). https://doi.org/10.1007/11586012_16

    Chapter  Google Scholar 

  51. Dunsmore, A., Roper, M., Wood, M.: The role of comprehension in software inspection. J. Syst. Softw. 52(2–3), 121–129 (2000)

    Article  Google Scholar 

  52. Da Cunha, A.D., Greathead, D.: Does personality matter?: an analysis of code-review ability. Commun. ACM 50(5), 109–112 (2007)

    Article  Google Scholar 

  53. Thongtanunam, P., McIntosh, S., Hassan, A.E., Iida, H.: Investigating code review practices in defective files: an empirical study of the qt system. In: Proceedings of the 12th Working Conference on Mining Software Repositories, pp. 168–179. IEEE (2015)

    Google Scholar 

  54. Kononenko, O., Baysal, O., Guerrouj, L., Cao, Y., Godfrey, M.W.: Investigating code review quality: do people and participation matter? In: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 111–120. IEEE (2015)

    Google Scholar 

  55. de Mello, R.M., Oliveira, R.F., Garcia, A.F.: On the influence of human factors for identifying code smells: a multi-trial empirical study. In: ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), vol. 2017, pp. 68–77. IEEE (2017)

    Google Scholar 

  56. Murakami, Y., Tsunoda, M., Uwano, H.: Wap: does reviewer age affect code review performance? In: IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), vol. 2017, pp. 164–169. IEEE (2017)

    Google Scholar 

  57. Seaman, C.B., Basili, V.R.: An empirical study of communication in code inspections. In: Proceedings of the (19th) International Conference on Software Engineering, pp. 96–106. IEEE (1997)

    Google Scholar 

  58. Porter, A., Siy, H., Mockus, A., Votta, L.: Understanding the sources of variation in software inspections. ACM Trans. Softw. Eng. Methodol. 7(1), 41–79 (1998)

    Article  Google Scholar 

  59. Müller, M.M.: Two controlled experiments concerning the comparison of pair programming to peer review. J. Syst. Softw. 78(2), 166–179 (2005)

    Article  Google Scholar 

  60. Spohrer, K., Kude, T., Schmidt, C.T., Heinzl, A.: Knowledge creation in information systems development teams: The role of pair programming and peer code review. In: ECIS. 213 (2013)

    Google Scholar 

  61. Miller, J., Yin, Z.: A cognitive-based mechanism for constructing software inspection teams. IEEE Trans. Softw. Eng. 30(11), 811–825 (2004)

    Article  Google Scholar 

  62. Sutherland, A., Venolia, G.: Can peer code reviews be exploited for later information needs? In: 2009 31st International Conference on Software Engineering-Companion Volume, pp. 259–262. IEEE (2009)

    Google Scholar 

  63. Seaman, C.B., Basili, V.R.: Communication and organization: An empirical study of discussion in inspection meetings. IEEE Trans. Softw. Eng. 24(7), 559–572 (1998)

    Article  Google Scholar 

  64. Bosu, A., Carver, J.C., Bird, C., Orbeck, J., Chockley, C.: Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Trans. Softw. Eng. 43(1), 56–75 (2016)

    Article  Google Scholar 

  65. Panichella, S., Arnaoudova, V., Di Penta, M., Antoniol, G.: Would static analysis tools help developers with code reviews? In: IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 161–170. IEEE (2015)

    Google Scholar 

  66. Thompson, C., Wagner, D.: A large-scale study of modern code review and security in open source projects. In: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering, pp. 83–92. ACM (2017)

    Google Scholar 

  67. Lei, Q., He, Z., Fuqun, H., Bin, L.: Classification of air on-board software code defects and investigations. Procedia Eng. 15, 3577–3583 (2011)

    Article  Google Scholar 

  68. Russell, G.W.: Experience with inspection in ultralarge-scale development. IEEE Softw. 8(1), 25–31 (1991)

    Article  Google Scholar 

  69. Siy, H., Votta, L.: Does the modern code inspection have value? In: Proceedings of the IEEE International Conference on Software- Maintenance (ICSM 2001), p. 281. IEEE (2001)

    Google Scholar 

  70. Bavota, G., Russo, B.: Four eyes are better than two: On the impact of code reviews on software quality. In: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 81–90. IEEE (2015)

    Google Scholar 

  71. Bosu, A., Carver, J.C.: Impact of peer code review on peer impression formation: a survey. In: ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, vol. 2013, p. 133–142. IEEE (2013)

    Google Scholar 

  72. Beller, M., Bacchelli, A., Zaidman, A., Juergens, E.: Modern code reviews in open-source projects: which problems do they fix? In: Proceedings of the 11th Working Conference on Mining Software Repositories, pp. 202–211. ACM (2014)

    Google Scholar 

  73. Bernhart, M., Grechenig, T.: On the understanding of programs with continuous code reviews. In: 2013 21st International Conference on Program Comprehension (ICPC), pp. 192–198. IEEE (2013)

    Google Scholar 

  74. Mäntylä, M.V., Lassenius, C.: What types of defects are really discovered in code reviews? IEEE Trans. Softw. Eng. 35(3), 430–448 (2008)

    Article  Google Scholar 

  75. Runeson, P., Stefik, A., Andrews, A.: Variation factors in the design and analysis of replicated controlled experiments. Empir. Softw. Eng. 19(6), 1781–1808 (2014)

    Article  Google Scholar 

  76. Baysal, O., Kononenko, O., Holmes, R., Godfrey, M.W.: Investigating technical and non-technical factors influencing modern code review. Empir. Softw. Eng. 21(3), 932–959 (2016)

    Article  Google Scholar 

  77. Nanthaamornphong, A., Chaisutanon, A.: Empirical evaluation of code smells in open source projects: preliminary results. In: Proceedings of the 1st International Workshop on Software Refactoring, pp. 5–8. ACM (2016)

    Google Scholar 

  78. Perry, D.E., Porter, A., Wade, M.W., Votta, L.G., Perpich, J.: Reducing inspection interval in large-scale software development. IEEE Trans. Softw. Eng. 28(7), 695–705 (2002)

    Article  Google Scholar 

  79. Jenkins, G.L., Ademoye, O.: Can individual code reviews improve solo programming on an introductory course? Innov. Teach. Learn. Inf. Comput. Sci. 11(1), 71–79 (2012)

    Google Scholar 

  80. Baum, T., Liskin, O., Niklas, K., Schneider, K.: Factors influencing code review processes in industry. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 85–96. ACM (2016)

    Google Scholar 

  81. Ackerman, A.F., Buchwald, L.S., Lewski, F.H.: Software inspections: an effective verification process. IEEE Softw. 6(3), 31–36 (1989)

    Article  Google Scholar 

  82. Rigby, P., Cleary, B., Painchaud, F., Storey, M.A., German, D.: Contemporary peer review in action: lessons from open source development. IEEE Softw. 29(6), 56–61 (2012)

    Article  Google Scholar 

  83. Kitchenham, B., Brereton, P.: A systematic review of systematic review process research in software engineering. Inf. Softw. Technol. 55(12), 2049–2075 (2013)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Free University of Bozen-Bolzano, Bozen-Bolzano, Italy

    Ilenia Fronza

  2. University of Helsinki, Helsinki, Finland

    Arto Hellas, Petri Ihantola & Tommi Mikkonen

Authors
  1. Ilenia Fronza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arto Hellas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Petri Ihantola
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tommi Mikkonen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tommi Mikkonen .

Editor information

Editors and Affiliations

  1. Vienna University of Technology, Vienna, Austria

    Dietmar Winkler

  2. Vienna University of Technology, Vienna, Austria

    Stefan Biffl

  3. fortiss GmbH, Germany, and Blekinge Institute of Technology, Karlskrona, Sweden

    Daniel Mendez

  4. Software Quality Lab GmbH, Linz, Austria

    Johannes Bergsmann

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fronza, I., Hellas, A., Ihantola, P., Mikkonen, T. (2020). Code Reviews, Software Inspections, and Code Walkthroughs: Systematic Mapping Study of Research Topics. In: Winkler, D., Biffl, S., Mendez, D., Bergsmann, J. (eds) Software Quality: Quality Intelligence in Software and Systems Engineering. SWQD 2020. Lecture Notes in Business Information Processing, vol 371. Springer, Cham. https://doi.org/10.1007/978-3-030-35510-4_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-35510-4_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-35509-8

  • Online ISBN: 978-3-030-35510-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation