skip to main content
10.1145/1414004.1414031acmconferencesArticle/Chapter ViewAbstractPublication PagesesemConference Proceedingsconference-collections
research-article

Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects

Published: 09 October 2008 Publication History

Abstract

Project managers can use capture-recapture models to manage the inspection process by estimating the number of defects present in an artifact and determining whether a reinspection is necessary. Researchers have previously evaluated capture-recapture models on artifacts with a known number of defects. Before applying capture-recapture models in real development, an evaluation of those models on naturally-occurring defects is imperative. The data in this study is drawn from two inspections of real requirements documents (that later guided implementation) created as part of a capstone course (i.e. with naturally occurring defects). The major results show that: a) estimators improve from being negatively biased after one inspection to being positively biased after two inspections, b) the results contradict the earlier result that a model that includes two sources of variation is a significant improvement over models with one source of variation, and c) estimates are useful in determining the need for artifact reinspection.

References

[1]
Ackerman, A., Buchwald, L., and Lewski, F., "Software Inspections: An Effective Verification Process." IEEE Software, 1989. 6(3): 31--36.
[2]
Briand, L. C., Emam, K. E., Freimut, B. G., and Laitenberger, O., "A Comprehensive Evaluation of Capture Recapture Models for Estimating Software Defect Content." IEEE Transactions on Software Engineering, 2000. 26(6): 518--539.
[3]
Burnham, K. P. and Overtom, W. S., "Estimation of the Size of a Closed Population When Capture Probabilities Vary among Animals." Biometrics, 1978. 65:625--633.
[4]
Chao, A., "Estimation the population Size for Capture-Recapture Data with Unequal Catchability." Biometrics, 1987. 43(4): 783--791.
[5]
Chao, A., "Estimating Animal Abundance with Capture Frequency Data." Journal of Wildlife Management, 1988. 52(2): 295--300
[6]
Chao, A. and Yeng, H. C., Program CARE-2 (for Capture-Recapture Part.2), http://chao.stat.nthu.edu.tw
[7]
Darroch, J. N., "The Multiple-Recapture Conensus 1: Estimation of a Closed Population." Biometrika, 1958. 45: 343--359.
[8]
Eick, S., Loader, C., Long, M., Votta, L., and Weil, S. V. "Estimating Software Fault Content Before Coding". In Proceedings of the 14th International Conference on Software Engineering. 1992. Melbourne, Australia: ACM Press: 59--65.
[9]
Eick, S., Loader, C., Weil, S. V., and Votta, L. "How Many Errors Remain in a Software Design after Inspection". In Proceedings of the 25th Symposium on the Interface. 1993.
[10]
El-Emam, K., Laitenberger, O., and Harbrich, T., "The Application of Subjective Estimates of Effectiveness to Controlling Software Inspections" Journal of Systems and Software, 2000. 54(2): 119--136.
[11]
El-Emam, K. and Laitenberger, O., "Evaluating Capture-Recapture Models with Two Inspectors." IEEE Transactions on Software Engineering, 2001. 27(9): 851--864
[12]
Lee, S. M. and Chao, A., "Estimating Population Size via Sample Coverage for Closed Capture-Recapture Models." Biometrics, 1994. 50: 88--97.
[13]
Miller, J., "Estimating the Number of Remaining Defects after Inspection." Software Testing, Verification and Reliability, 1999. 9(3): 167--189.
[14]
Musa, J., Iannion, A., Okumoto, O., "Software Reliability: Measurement, Predicition, Application," McGraw-Hill, 1987
[15]
Otis, D., Burnham, K., White, G., and Anderson, D., "Statistical Inference from Capture Data on Closed Animal Population." Wildlife Monograph, 1978. 64: 1--135.
[16]
Petersson, H., Thelin, T., Runeson, P., Wohlin, C. "Capture-Recapture in Software Inspections after 10 Years Research - Theory, Evaluation, and Application." The Journal of Systems and Software, 72(2):249--264.
[17]
Runeson, P. and Wohlin, C., "An Experimental Evaluation of an Experience-Based Capture-Recapture Method in Software Code Inspections." Empirical Software Engineering: An International Journal, 1998. 3(4): 381--406.
[18]
Thelin, T., Petersson, P., and Runeson, P., "Confidence Intervals for Capture-Recapture Estimations in Software Inspections." Journal of Information and Software Technology, 2002. 44(12): 683--702.
[19]
Walia, G. S. and Carver, J. C., Philip, T., "Requirement Error Abstraction and Classiifcation: An Empirical Study," In Proceedings of 5th International Symposium on Empirical Software Engineering, Rio de Janerio, 2006, pp. 336--345.
[20]
Walia, G., Carver, J., and Nagappan, N. "The Effect of the Number of Inspectors on the Defect Estimates Produced by Capture-Recapture Models." To appear in the Proceedings of the 30th International Conference in Software Engineering. May 10-18, 2008. Leipzig, Germany.
[21]
Weil, S. V. and Votta, L., "Assessing Software Designs Using Capture-Recapture Methoda." IEEE Transactions on Software Engineering, 1993. 19(11): 1045--1054.
[22]
White, G. C., Anderson, D. R., Burnham, K. p., and Otis, D. l., Capture-Recapture and Removal Methods for Sampling Closed Populations, Los Alamos National Laboratory, 1982.
[23]
Wohlin, C., Runeson, P., and Brantestam, J., "An Experimental Evaluation of Capture-Recapture in Software Inspections." Software Testing, Verification and Reliability, 1995. 5(4): 213--232.
[24]
Wohlin, C. and Runeson, P. "Defect Content Estimation from Review Data". In Proceedings of the 20th International Conference on Software Engineering. 1998. Kyoto, Japan: IEEE Computer Society Press: 400--409.
[25]
Yip, P. S. F., "A Martingale Estimating Equation for a Capture-Recapture Experiment in Discrete Time." Biometrics, 1991. 47: 1081--1088.

Cited By

View all
  • (2018)An empirical study of software change classification with imbalance data‐handling methodsSoftware: Practice and Experience10.1002/spe.260648:11(1968-1999)Online publication date: 29-Jun-2018
  • (2017)Towards Confidence with Capture-recapture EstimationProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering10.1145/3084226.3084250(242-251)Online publication date: 15-Jun-2017
  • (2016)What sample sizes do we need? Part 2: formative studiesQuantifying the User Experience10.1016/B978-0-12-802308-2.00007-2(143-183)Online publication date: 2016
  • Show More Cited By

Index Terms

  1. Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects

                      Recommendations

                      Comments

                      Information & Contributors

                      Information

                      Published In

                      cover image ACM Conferences
                      ESEM '08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
                      October 2008
                      374 pages
                      ISBN:9781595939715
                      DOI:10.1145/1414004
                      • General Chair:
                      • Dieter Rombach,
                      • Program Chairs:
                      • Sebastian Elbaum,
                      • Jürgen Münch
                      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

                      Sponsors

                      Publisher

                      Association for Computing Machinery

                      New York, NY, United States

                      Publication History

                      Published: 09 October 2008

                      Permissions

                      Request permissions for this article.

                      Check for updates

                      Author Tags

                      1. capture-recapture models
                      2. defect estimation
                      3. empirical study
                      4. requirements
                      5. software inspections
                      6. validation and verification

                      Qualifiers

                      • Research-article

                      Conference

                      ESEM '08
                      Sponsor:

                      Acceptance Rates

                      Overall Acceptance Rate 130 of 594 submissions, 22%

                      Contributors

                      Other Metrics

                      Bibliometrics & Citations

                      Bibliometrics

                      Article Metrics

                      • Downloads (Last 12 months)0
                      • Downloads (Last 6 weeks)0
                      Reflects downloads up to 17 Feb 2025

                      Other Metrics

                      Citations

                      Cited By

                      View all
                      • (2018)An empirical study of software change classification with imbalance data‐handling methodsSoftware: Practice and Experience10.1002/spe.260648:11(1968-1999)Online publication date: 29-Jun-2018
                      • (2017)Towards Confidence with Capture-recapture EstimationProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering10.1145/3084226.3084250(242-251)Online publication date: 15-Jun-2017
                      • (2016)What sample sizes do we need? Part 2: formative studiesQuantifying the User Experience10.1016/B978-0-12-802308-2.00007-2(143-183)Online publication date: 2016
                      • (2015)An Empirical Evaluation of Capture-Recapture Estimators in Software InspectionProceedings of the 2015 24th Australasian Software Engineering Conference (ASWEC)10.1109/ASWEC.2015.17(58-67)Online publication date: 28-Sep-2015
                      • (2014)Dealing with Completeness in Requirements EngineeringEncyclopedia of Information Science and Technology, Third Edition10.4018/978-1-4666-5888-2.ch279(2854-2863)Online publication date: 31-Jul-2014
                      • (2014)An Overview of Experimental Studies on Software Inspection ProcessEnterprise Information Systems10.1007/978-3-319-09492-2_8(118-134)Online publication date: 25-Jul-2014
                      • (2013)Using capture-recapture models to make objective post-inspection decisions2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW.2013.6688868(54-54)Online publication date: Nov-2013
                      • (2012)Application of kusumoto cost-metric to evaluate the cost effectiveness of software inspectionsProceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement10.1145/2372251.2372291(221-230)Online publication date: 19-Sep-2012
                      • (2012)Sample size in usability studiesCommunications of the ACM10.1145/2133806.213382455:4(64-70)Online publication date: 1-Apr-2012
                      • (2012)Checklist-Based Inspection Technique for Feature Models ReviewProceedings of the 2012 Sixth Brazilian Symposium on Software Components, Architectures and Reuse10.1109/SBCARS.2012.25(140-149)Online publication date: 23-Sep-2012
                      • Show More Cited By

                      View Options

                      Login options

                      View options

                      PDF

                      View or Download as a PDF file.

                      PDF

                      eReader

                      View online with eReader.

                      eReader

                      Figures

                      Tables

                      Media

                      Share

                      Share

                      Share this Publication link

                      Share on social media