research-article

Code coverage analysis in practice for large systems

Authors:

Nadav Steindler,

Aviad ZlotnickAuthors Info & Claims

ICSE '11: Proceedings of the 33rd International Conference on Software Engineering

Pages 736 - 745

https://doi.org/10.1145/1985793.1985897

Published: 21 May 2011 Publication History

Abstract

Large systems generate immense quantities of code coverage data. A user faced with the task of analyzing this data, for example, to decide on test areas to improve, faces a 'needle in a haystack' problem.

In earlier studies we introduced substring hole analysis, a technique for presenting large quantities of coverage data in a succinct way. Here we demonstrate the successful use of substring hole analysis on large scale data from industrial software systems. For this end we augment substring hole analysis by introducing a work flow and tool support for practical code coverage analysis.

We conduct real data experiments indicating that augmented substring hole analysis enables code coverage analysis where it was previously impractical, correctly identifies functionality that is missing from existing tests, and can increase the probability of finding bugs. These facilitate cost-effective code coverage analysis.

References

[1]

Yoram Adler, Eitan Farchi, Moshe Klausner, Dan Pelleg, Orna Raz, Moran Shochat, Shmuel Ur, and Aviad Zlotnick. Advanced code coverage analysis using substring holes. In ISSTA '09: Proceedings of the 2009 ACM SIGSOFT international symposium on software testing and analysis, 2009.

Digital Library

[2]

Yoram Adler, Eitan Farchi, Moshe Klausner, Dan Pelleg, Orna Raz, Moran Shochat, Shmuel Ur, and Aviad Zlotnick. Automated substring hole analysis. In Proceedings of the 2009 International Conference of Software Engineering, ICSE 2009, pages 203--206, 2009.

[3]

Boris Beizer. Software testing techniques (2nd ed.). Van Nostrand Reinhold Co., New York, NY, USA, 1990.

Digital Library

[4]

Arkady Bron, Eitan Farchi, Yonit Magid, Yarden Nir, and Shmuel Ur. Applications of synchronization coverage. In PPOPP, pages 206--212, 2005.

Digital Library

[5]

Marc Eaddy, Thomas Zimmermann, Kaitlin D. Sherwood, Vibhav Garg, Gail C. Murphy, Nachiappan Nagappan, and Alfred V. Aho. Do crosscutting concerns cause defects? IEEE Transactions on Software Engineering, 34(4):497--515, July 2008.

Digital Library

[6]

S. Elbaum, A. G. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE Transactions on Software Engineering, 28(2):159--182, 2002.

Digital Library

[7]

FoCuS. http://www.alphaworks.ibm.com/tech/focus.

[8]

Todd L. Graves, Mary Jean Harrold, Jung-Min Kim, Adam Porter, and Gregg Rothermel. An empirical study of regression test selection techniques. ACM Trans. Softw. Eng. Methodol., 10(2):184--208, 2001.

Digital Library

[9]

Yong Woo Kim. Efficient use of code coverage in large-scale software development. In CASCON '03: Proceedings of the 2003 conference of the Centre for Advanced Studies on Collaborative research, pages 145--155. IBM Press, 2003.

Digital Library

[10]

Brian Marick. How to misuse code coverage. In Proceedings of the 16th Interational Conference on Testing Computer Software, pages 16--18, 1999.

[11]

Christina Pavlopoulou and Michal Young. Residual test coverage monitoring. In ICSE '99: Proceedings of the 21st international conference on Software engineering, pages 277--284, New York, NY, USA, 1999. ACM.

Digital Library

[12]

PHP hypertext preprocessor. http://www.php.net/.

[13]

Orna Raz, Moshe Klausner, Nitzan Peleg, Gadi Haber, Eitan Farchi, Shachar Fienblit, Yakov S. Filiarsky, Shay Gammer, and Sergey Novikov. The advantages of post-link code coverage. In Haifa Verification Conference, pages 235--251, 2007.

Digital Library

[14]

Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27:929--948, 2001.

Digital Library

[15]

Mustafa M. Tikir and Jeffrey K. Hollingsworth. Efficient instrumentation for code coverage testing. In ISSTA '02: Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, pages 86--96, New York, NY, USA, 2002. ACM.

Digital Library

Cited By

Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
van Heijningen SWiik TNeto FGay GViggedal KFriberg DFilkov VRay BZhou M(2024)Integrating Mutation Testing Into Developer Workflow: An Industrial Case StudyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695273(2110-2120)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695273
Liu YAhuja GKuenning GSmolka SZadok ETarasov VZhang YAnwar AMi N(2023)Input and Output Coverage Needed in File System TestingProceedings of the 15th ACM Workshop on Hot Topics in Storage and File Systems10.1145/3599691.3603405(93-101)Online publication date: 9-Jul-2023
https://dl.acm.org/doi/10.1145/3599691.3603405
Show More Cited By

Index Terms

Code coverage analysis in practice for large systems
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Code coverage for suite evaluation by developers
ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

One of the key challenges of developers testing code is determining a test suite's quality -- its ability to find faults. The most common approach is to use code coverage as a measure for test suite quality, and diminishing returns in coverage or high ...
Efficient Incremental Code Coverage Analysis for Regression Test Suites
ASE '24: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

Code coverage analysis has been widely adopted in the continuous integration of open-source and industry software repositories to monitor the adequacy of regression test suites. However, computing code coverage can be costly, introducing significant ...
On the Relationship between Model Coverage and Code Coverage Using MATLAB's Simulink
QRS '15: Proceedings of the 2015 IEEE International Conference on Software Quality, Reliability and Security

Software Testing is an approach to ensuring the quality of software systems. Testing of safety-critical systems often requires conformance to certain code coverage criteria, including for example, in aviation, Modified Condition/Decision Coverage (MC/DC)...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '11: Proceedings of the 33rd International Conference on Software Engineering

May 2011

1258 pages

ISBN:9781450304450

DOI:10.1145/1985793

General Chair:
Richard N. Taylor
UC Irvine, USA
,
Program Chairs:
Harald Gall
University of Zurich, Switzerland
,
Nenad Medvidović
University of Southern California, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 May 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE11

Sponsor:

SIGSOFT

ICSE11: International Conference on Software Engineering

May 21 - 28, 2011

HI, Waikiki, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
509
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)2

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
van Heijningen SWiik TNeto FGay GViggedal KFriberg DFilkov VRay BZhou M(2024)Integrating Mutation Testing Into Developer Workflow: An Industrial Case StudyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695273(2110-2120)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695273
Liu YAhuja GKuenning GSmolka SZadok ETarasov VZhang YAnwar AMi N(2023)Input and Output Coverage Needed in File System TestingProceedings of the 15th ACM Workshop on Hot Topics in Storage and File Systems10.1145/3599691.3603405(93-101)Online publication date: 9-Jul-2023
https://dl.acm.org/doi/10.1145/3599691.3603405
Ivanković MPetrović GJust RFraser GDumas MPfahl DApel SRusso A(2019)Code coverage at GoogleProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3340459(955-963)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3340459
Chen BSong JXu PHu XJiang ZHuchard MKästner CFraser G(2018)An automated approach to estimating code coverage measures via execution logsProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering10.1145/3238147.3238214(305-316)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3238147.3238214
Zheng HLi DLiang BZeng XZheng WDeng YLam WYang WXie TJuristo NShepherd D(2017)Automated test input generation for AndroidProceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track10.1109/ICSE-SEIP.2017.32(253-262)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICSE-SEIP.2017.32
Avros RDudka VKřena BLetko ZPluháčková HUr SVojnar TVolkovich Z(2017)Boosted decision trees for behaviour mining of concurrent programmesConcurrency and Computation: Practice and Experience10.1002/cpe.426829:21Online publication date: 31-Aug-2017
https://doi.org/10.1002/cpe.4268
Patil ASidnal N(2014)CodeCoverACM SIGSOFT Software Engineering Notes10.1145/2557833.255785039:1(1-4)Online publication date: 11-Feb-2014
https://dl.acm.org/doi/10.1145/2557833.2557850

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten