Skip to main content
SpringerLink
Log in

A binomial software reliability model based on coverage of structural testing criteria

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

A new approach to software reliability modeling is discussed where variables indirectly related with software reliability are used to provide additional information for the modeling process. Previous studies, empirical and theoretical evidences, and results from experiments indicate that there is a strong relationship between software reliability and coverage of program elements required to be exercised by structural testing criteria. This paper develops a binomial type coverage-based software reliability model through the definition of a coverage-based failure rate function. The Binomial software reliability Model Based on Coverage—BMBC—is proposed and discussed. In the BMBC test data between failures is used instead of time as independent variable; the model was assessed with test data from a real application, making use of the following structural testing criteria: all-nodes, all-edges, and potential-uses—a data-flow based family of testing criteria. The results from our experiments have shown that our modeling approach has some advantages over some traditional reliability models and points to a very promising research direction in software reliability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

C j :

testing criterion

c j :

measured coverage for testing criterion C j

1 − c j :

complement of measured coverage for testing criterion Cj

Z a (n):

failure rate function for n test data and failure “a”

α :

complement of normalized coverage for n test data

N :

initial number of faults in the software

n i :

accumulated number of test data for ith failure (i = 1, 2, 3, ..., N)

α i :

complement of normalized coverage for n i test data

k i :

number of test data between the ith and (i + 1)th failures

λ (k i |n i ):

failure rate conditioned to the remaining faults in the software

R(k i |n i ):

conditional reliability

c ji :

measured coverage for testing criterion C j after the ith failure

y ji :

1 − c ji complement of measured coverage c ji

S j :

number of elements required by testing criterion C j

s ji :

number of elements required by testing criterion C j exercised by n i test data

n :

number of executions of the software

n e :

number of executions of the software with failure

References

  • Adams EN (1980) Minimizing Cost Impact of Software Defects, IBM Research Division, Report RC 8228(35669)

  • Bastani FB, Ramamoorthy CV (1986) Input-domain-based Models for Estimating the Correctness of Process Control Programs. In Serra A, Barlow RE (eds), Reliability Theory. North Holland, Amsterdam, pp 321–378

    Google Scholar 

  • Beizer B (1990) Software testing techniques, 2nd ed. Van Nostrand Rheinhold, New York

    Google Scholar 

  • Bishop P (ed.) (1990) Prediction and measurement of software reliability, dependability of critical computer systems 3. Elsevier, London

  • Butler RW, Finelli GB (1991) The Unfeasibility of Experimental Quantification of Life-critical Software Reliability. Proceedings of the ACM SIGSOFT ‘91 Conference on Software for Critical Systems, ACM Press, pp 66–76

  • Chaim ML (1991) Uma Ferramenta para Suporte ao Teste Estrutural de Programas Baseado em Análise de Fluxo de Dados. Master Thesis. DCA/FEE/UNICAMP-Campinas, SP-Brazil (in Portuguese)

  • Chen M (1994) Tools and Techniques for Testing Based Software Reliability Estimation. PhD Dissertation. Purdue University, West Lafayette–Indiana

  • Chen M, Mathur AP, Rego VJ (1995) Effect of testing technique on software reliability estimates obtained using a time-domain model. IEEE Trans Reliab 44(1):97–103

    Article  Google Scholar 

  • Chen M, Lyu MR, Wong WE (1996) An empirical study of the correlation between code coverage and reliability estimation. Proceedings of the Third International Software Metrics Symposium, Berlin, Germany

  • Chen M, Lyu MR, Wong WE (1997) Incorporating code coverage in the reliability estimation for fault-tolerant software, Proceedings of the Symposium on Reliable Distributed Systems, Durham, NC

  • Chen M, Lyu MR, Wong WE (2001) Effect od code coverage on software reliability measurement. IEEE Trans Reliab 50(2):165–170

    Article  Google Scholar 

  • Crespo AN (1997) Modelos de Confiabilidade de Software Baseados em Cobertura de Critérios Estruturais de Teste. Tese de Doutorado, DCA/FEEC/UNICAMP—Campinas, SP, Brazil (in Portuguese)

  • Crespo AN, Matrella P, Pasquini A (1996) Sensitivity of reliability growth models to operational profile errors. Proceedings, The Seventh International Symposium on Software Reliability Engineering, White Plains, New York, pp 35–44

  • Crespo AN, Jino M, Pasquini A, Maldonado JC (1997) Cobertura dos Critérios Potenciais-usos e Confiabilidade do Software. Anais do XI Simpósio Brasileiro de Engenharia de Software—Fortaleza, Ceará-Brazil (in Portuguese)

  • Farr WH, Smith OD (1993) Statistical Modeling and Estimation of Reliability Functions for Software (SMERFS) User’s Guide, Naval Surface Warfare Center, Virginia

  • Frankl PG, Weyuker EJ (1988) An applicable family of data flow testing criteria. IEEE Trans Softw Eng SE-14(10):1483–1498

    Article  MathSciNet  Google Scholar 

  • Frate FD, Garg P, Mathur AP, Pasquini A (1995a) Experiments to Investigate the Correlation Between Code Coverage and Software Reliability. Software Engineering Research Center, Purdue University, West Lafayette, Indiana, SERC-TR-162-P

  • Frate FD, Garg P, Mathur AP, Pasquini A (1995b) On the correlation between code coverage and software reliability. Proceedings of The Sixth International Symposium on Software Reliability Engineering, Toulouse, France, pp 124–132

  • Garg P (1994) Investigating coverage—reliability relationship and sensitivity of reliability to errors in operational profile. Department of Computer Sciences, Purdue University, West Lafayette, Indiana, Technical Report

  • Goel AL, Okumoto K (1979) A time dependent error detection rate model for software reliability and other performance measures. IEEE Trans Reliab 28:206–211

    Article  MATH  Google Scholar 

  • Gokhale SS, Philip T, Marinos PN, Trivedi KS (1996) Unification of finite failure non-homogeneous Poisson process models through test coverage. Proceedings of The Seventh International Symposium on Software Reliability Engineering, White Plains, NY, USA, pp 289–299

  • Grottke M (2002) A vector markov model for structural coverage growth and the number of failure occurrences, Proceedings of the 13th International Symposium on Software Reliability Engineering—ISSRE

  • Hamlet D (1992) Are we testing for True Reliability, IEEE Softw 9(4)

  • Horgan JR, Mathur AP, Pasquini A, Rego VJ (1995) Perils of software reliability modeling, Technical Report SERC-TR-160-P, Purdue University

  • Hudson A (1967) Program errors as a birth and death process. Systems Development Corporation, Santa Monica, CA, Technical Report SP-3011

  • ISO/IEC (1991) ISO/IEC 9126—Information technology—Software product evaluation—quality characteristics and guidelines for their use. ISO/IEC, International Standard

  • Jalote P, Muralidhara YR (1994) A coverage based model for software reliability estimation. Proceedings of the First International Conference on Software Testing, Reliability and Quality Assurance, New Delhi, Índia, pp 6–10

  • Jelinski Z, Moranda PB (1972) Software reliability research. Proceedings of the Statistical Methods for the Evaluation of Computer System Performance Academic Press pp 465–484

  • Krishnamurthy S, Mathur AP (1996) On predicting reliability of modules using code coverage. Proceedings of the Conference of the Centre for Advanced Studies on Collaborative Research, Toronto, Canada, pp 1–12

  • Li M, Wei Y, Desovski D, Nejad H, Ghose S, Cukic B, Smidts C (2004) Validation of a methodology for assessing software reliability. Proceedings of the 15th International Symposium on Software Reliability Engineering, Saint-Malo, France, pp 66–76

  • Littlewood B, Strigini L (1993) Validation of ultrahigh dependability for software-based systems. Communications of the ACM 36(1)

  • Lyu MR (1996) Handbook of software reliability engineering. McGraw-Hill, New York

    Google Scholar 

  • Malaiya YK, Li N, Bieman J, Karcich R, Skibbe B (1994) The relationship between test coverage and reliability. Proceedings of the Fifth International Symposium on Software Reliability Engineering, Monterey, CA, pp 186–195

  • Malaiya YK, Li N, Bieman J, Karcich R, Skibbe B (1996) Software test coverage and reliability, technical report CS-96-128, Colorado State University

  • Malaiya YK, Li N, Bieman J, Karcich R (2002) Software reliability growth with test coverage. IEEE Trans Reliab 51(4):420–426

    Article  Google Scholar 

  • Maldonado JC, Chaim ML, Jino M (1992a) Bridging the gap in the presence of infeasible paths: Potential uses testing criteria. XII International Conference of the SCCC, Sociedad Chilena de Ciencia de la Computacion, Santiago, Chile, pp 323–340

  • Maldonado JC, Chaim ML, Jino M (1992b) Using the essencial branch concept to support data-flow based testing criteria application. Fifth International Conference on Software Engineering and Its Applications; Toulouse, France, pp 613–623

  • Mood A, Graybill F, Boes D (1974) Introduction to the theory of statistics, 3d ed. McGraw-Hill, New York

    MATH  Google Scholar 

  • Musa JD (1999) Software reliability engineering. McGraw-Hill, New York

    Google Scholar 

  • Musa JD, Ianino A, Okumoto K (1987) Software reliability—measurement, prediction, application. McGraw-Hill, New York

    Google Scholar 

  • Myers GJ (1979) The art of software testing. Wiley, New York

    Google Scholar 

  • Nelson F (1978) Estimating software reliability from test data. Microelectron Reliab 17(1):67–73

    Article  Google Scholar 

  • Pham H, Zhang X (2003) NHPP Software reliability and cost models with testing coverage. Eur J Oper Res 145:443–454

    Article  MATH  Google Scholar 

  • Pressman RS (2001) Software engineering—a practitioner’s approach, 5th ed. McGraw-Hill

  • Ramsey J, Basili VR (1985) Analyzing the test process using structural coverage. Proceedings of the ICSE, pp 306–312

  • Rapps S, Weyuker EJ (1985) Selecting software test data using data flow information. IEEE Trans Softw Eng SE-11(4):367–375

    Article  Google Scholar 

  • Shooman ML (1972) Probabilistic models for software reliability prediction. In Freiberg W (ed), Statistical computer performance evaluation. New York, Academic Press, pp 485–502

    Google Scholar 

  • Siegel S (1956) Nonparametric Statistics for the Behavioral Sciences. McGraw Hill Kogakusha

  • Varadan GS (1995) Trends in reliability and test strategies. IEEE Softw 10

  • Veevers A, Marshall A (1994) A relationship between software coverage metrics and reliability. Softw Test Verif Reliab 4:3–8

    Article  Google Scholar 

  • Xie M (1993) Software reliability models—a selected annotated bibliography. Softw Test Verif Reliab 3:3–28

    Article  Google Scholar 

  • Ye R, Malaiya YK (2002) Relationship between test effectiveness and coverage. Proceedings of the 13th International Symposium on Software Reliability Engineering—ISSRE

Download references

Acknowledgements

The authors thank the anonymous referees and the editor for their valuable suggestions and comments which helped to improve the paper. Thanks are also due to Marcos Lordello Chaim who implemented Poke-Tool and helped to run the experiment. ENEA, Rome, Italy, hosted one author as a visiting researcher while part of the work was being developed. This work was partially supported by HP-Brazil.

Author information

Authors and Affiliations

  1. Centro de Pesquisas Renato Archer—CenPRA, Campinas, SP, Brazil

    Adalberto Nobiato Crespo

  2. Faculdade de Engenharia Elétrica e de Computação—Unicamp, Campinas, SP, Brazil

    Mario Jino

  3. DeepBlue, Rome, Italy

    Alberto Pasquini

  4. ICMC/USP, São Carlos, SP, Brazil

    José Carlos Maldonado

Authors
  1. Adalberto Nobiato Crespo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mario Jino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alberto Pasquini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José Carlos Maldonado
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adalberto Nobiato Crespo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Crespo, A.N., Jino, M., Pasquini, A. et al. A binomial software reliability model based on coverage of structural testing criteria. Empir Software Eng 13, 185–209 (2008). https://doi.org/10.1007/s10664-007-9055-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-007-9055-3

Keywords

Search

Navigation