Skip to main content
SpringerLink
Log in

Software Reliability Experimentation and Control

  • Software
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

This paper classifies software researches as theoretical researches, experimental researches, and engineering researches, and is mainly concerned with the experimental researches with focus on software reliability experimentation and control. The state-of-the-art of experimental or empirical studies is reviewed. A new experimentation methodology is proposed, which is largely theory discovering oriented. Several unexpected results of experimental studies are presented to justify the importance of software reliability experimentation and control. Finally, a few topics that deserve future investigation are identified.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sipser M. Introduction to the Theory of Computation. PWS Publishing Company, 1997.

  2. M.H. Halstead. Elements of Software Science. North-Holland. 1977.

  3. Hudson G R. Program errors as a birth and death process. System Development Corporation. Report SP-3011, Santa Monica, CA, 1967.

  4. Basili V, Shull F, Lanubile F. Using experiments to build a body of knowledge. In Proc. Ershov Memorial Conference, 1999, pp. 265–282.

  5. Jelinski Z, Moranda P B. Software Reliability Research. Statistical Computer Performance Evaluation, Greiberger W (ed.), Academic Press, 1972, pp. 465–484.

  6. Musa J D. A theory of software reliability and its application. IEEE Trans. Software Engineering, 1975, SE-1(3): 312–327.

    Google Scholar 

  7. Littlewood B, Verrall J. A Bayesian reliability growth model for computer software. Applied Statistics, 1973, 22(3): 332–346.

    Article  MathSciNet  Google Scholar 

  8. Goel A L, Okumoto K. A time dependent error detection rate model for a large scale software system. In Proc. the Third USA-Japan Computer Conference, San Francisco, USA, 1978, pp. 35–40.

  9. Nelson E C. Estimating software reliability from test data. Microelectronics and Reliability, 1978, 17(1): 67–74.

    Article  Google Scholar 

  10. Mills H D. On the statistical validation of computer program. FCS-72-6015, IBM Federal System Division, 1972.

  11. Avizienis A. Fault tolerant systems. IEEE Trans. Computer, 1976, C-25: 1304–1312.

    MathSciNet  Google Scholar 

  12. Randell B. System structure for software fault tolerance. IEEE Trans. Software Engineering, 1975, SE-1(2): 220–232.

    Google Scholar 

  13. Cai K Y, Wen C Y, Zhang M.L. A critical review on software reliability modeling. Reliability Engineering and System Safety, 1991, 32: 357–371.

    Google Scholar 

  14. Anderson T, Barret P A, Halliwell D N, Moulding M R. Software fault tolerance: An evaluation. IEEE Trans. Software Engineering, 1985, SE-11(12): 1502–1510.

    Google Scholar 

  15. Knight J C, Leveson N G. An experimental evaluation of the assumption of independence in multi-version programming. IEEE Trans. Software Engineering, 1986, SE-12(1): 96–109.

    Google Scholar 

  16. Butler R W, Finelli G B. The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Trans. Software Engineering, 1993, 19(1): 3–12.

    Google Scholar 

  17. Littlewood B, Strigini L, Validation of ultra-high dependability for software-based systems. Communications of the ACM, 1993, 36(11): 69–80.

  18. Reussner R H, Schmidt H W, Poernomo I H. Reliability prediction for component-based architectures. Journal of Systems and Software, 2003, 66: 241–252.

    Article  Google Scholar 

  19. Huynh T, Miller J. Further investigations into evaluating website reliability. In Proc. Int. Symp. Empirical Software Engineering, Noosa Heads, Australia, 2005, pp. 162–171.

  20. Xie M. Software Reliability Modeling. World Scientific, 1991.

  21. Cai K Y. Software Defect and Operational Profile Modeling. Academic Publishers, 1998.

  22. Cai K Y, Wang X Y. Towards a control-theoretical approach to software fault-tolerance. In Proc. the 4th Int. Conference on Quality Software, IEEE Computer Society Press, Braunschweig, Germany, 2004, pp. 198–205.

  23. Pretschner A et al. One evaluation of model-based testing and its automation. In Proc. Int. Conf. Software Engineering, St. Louis, MO, USA, 2005, pp. 302–401.

  24. E M Clarke Jr, O Grumberg, D A Peled. Model Checking. The MIT Press, 1999.

  25. Biffl B, Gutjahr W J. Using a reliability growth model to control software inspection. Empirical Software Engineering, 2002, 7: 257–284.

    Article  MATH  Google Scholar 

  26. Stringfellow C, Andrews A A. An empirical method for selecting software reliability growth models. Empirical Software Engineering, 2002, 7: 319–343.

    Article  MATH  Google Scholar 

  27. Henningsson K, Wohlin C. Assuring fault classification agreement—An empirical evaluation. In Proc. Int. Symp. Empirical Software Engineering, 2004, pp. 95–104.

  28. Curtis B. Measurement and experimentation in software engineering. Proc. the IEEE, 1980, 68(9): 1144–1157.

  29. Tichy W F, Lukowicz P, Prechelt L et al. Experimental evaluation in computer science: A quantitative study. Journal of Systems and Software, 1995, 28: 9–18.

    Article  Google Scholar 

  30. Tichy W F. Should computer scientists experiment more? Computer, May 1998, 31(5): 32–40.

    Article  Google Scholar 

  31. Perry D E, Porter A A, Votta L G. Empirical studies of software engineering: A roadmap. In Proc. ICSE Workshop on Future of Software Engineering, Edinburgh, UK, 2000, pp. 347–355.

  32. Harrison R, Badoo N, Barry E et al. Directions and methodologies for empirical software engineering research. Empirical Software Engineering, 1999, 4(4): 405–410.

    Article  Google Scholar 

  33. Sjoberg D I K, Hannay J E, Hansen O et al. A survey of controlled experiments in software engineering. IEEE Trans. Software Engineering, 2005, 31(9): 733–753.

    Article  Google Scholar 

  34. Fenton N, Pfleeger S L, Glass R L. Science and substance: A challenge to software engineering. IEEE Software, July 1994, pp. 88–95.

  35. Zelkowitz M V, Wallance D R. Experimental models for validating technology. IEEE Software, May 1998, pp. 23–31.

  36. Rothermel G, Elbaum S, Malishevsky A G et al. On test suite composition and cost-effective regression testing. ACM Trans. Software Engineering and Methodology, 2004, 13(3): 277–331.

    Article  Google Scholar 

  37. Juristo N, Moreno A M, Vegas S. Reviewing 25 years of testing technique experiments. Empirical Software Engineering, 2004, 9: 7–44.

    Article  Google Scholar 

  38. Zendler A. A preliminary software engineering theory as investigated by published experiments. Empirical Software Engineering, 2001, 6: 161–180.

    Article  MATH  Google Scholar 

  39. IEEE. IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990, 1990.

  40. Kitchenham B A, Pfleeger S L, Pickard L M et al. Preliminary guidelines for empirical research in software engineering. IEEE Trans. Software Engineering, 2002, 28(8): 721–734.

    Article  Google Scholar 

  41. Cater-Steel A, Toleman M, Rout T. Addressing the challenges of surveys in software engineering. In Proc. 2005 Int. Symp. Empirical Software Engineering, 2005, pp. 204–213.

  42. Jedlitschka A, Pfahl D. Reporting guidelines for controlled experiments in software engineering. In Proc. 2005 Int. Symp. Empirical Software Engineering, Noosa Heads, Australia, 2005, pp. 95–104.

  43. Do H, Elbaum S, Rothermel G. Infrastructure support for controlled experimentation with software testing and regression testing techniques. In Proc. 2004 Int. Symp. Empirical Software Engineering, Redondo Beach, CA, USA, 2004, pp. 60–70.

  44. Boehm B et al. Using empirical testbeds to accelerate technology maturity and transition: The SCRover experience. In Proc. 2004 Int. Symp. Empirical Software Engineering, Redondo Beach, CA, USA, 2004, pp. 117–126.

  45. Cai K Y, Chen L. Analyzing software science data with partial repeatability. Journal of Systems and Software, 2002, 63: 173–186.

    Article  Google Scholar 

  46. Xie M. Software Reliability Modeling. World Scientific, 1991.

  47. Cai K Y. Software Defect and Operational Profile Modeling. Kluwer Academic Publishers, 1998.

  48. Pham H. Software reliability and cost models: Perspectives, comparison, and practice. European Journal of Operational Research, 2003, 149: 475–489.

    Article  MATH  MathSciNet  Google Scholar 

  49. Vokolos F I, Frankl R H. Empirical evaluation of the textual differencing regression testing technique. In Proc. the Int. Conf. Software Maintenance, Bethesda, Maryland, USA, November 1998, pp. 44–53.

  50. Rothermel G, Untch R H, Chu C et al. Prioritizing test cases for regression testing. IEEE Trans. Software Engineering, 2001, 27(10): 929–948.

    Article  Google Scholar 

  51. Grassi V, Mirandola R. Devivation of Markov models for effectiveness analysis of adaptable software architectures for mobile computing. IEEE Trans. Mobile Computing, 2003, 4(2): 114–131.

    Article  Google Scholar 

  52. Goseva-Popstojanova K, Trivedi K. Architecture-based approach to reliability assessment of software systems. Performance Evaluation, 2001, 45: 179–204.

    Article  MATH  Google Scholar 

  53. Cai K Y, Lei X F, Shi Y et al. Partial-repeatability as a new form of uncertainty. Fuzzy Logic, Software Computing and Computational Intelligence, Yingming Liu, Guoqing Chen, Mingsheng Ying (eds.), Proc. 11th Int. Fuzzy Systems Association World Congress, Tsinghua University Press/Springer, Beijing, 2005, pp. 157–166.

  54. Whittaker J A. What is software testing? And why is it so hard? IEEE Software, January/February, 2000, 17(1): 70–79.

    Article  Google Scholar 

  55. Cai K Y. Optimal software testing and adaptive software testing in the context of software cybernetics. Information and Software Technology, 2002, 44: 841–855.

    Article  Google Scholar 

  56. Cai K Y, Li Y C, Liu K. Optimal and adaptive testing for software reliability assessment. Information and Software Technology, 2004, 46: 989–1000.

    Article  Google Scholar 

  57. Guo Y L, Shen F J. The History of Physics. Tsinghua University Press, 1993. (in Chinese)

  58. Johannes Kepler. His life, his laws and times. http://kepler.nasa.gov/johannes/.

  59. Li M, Vitanyi P M B. An Introduction to Kolmogorov Complexity and Its Applications, Second Edition. Springer-Verlag, 1997.

  60. Wiener N. Cybernetics: Or Control and Communication in the Animal and the Machine. John Wiley & Sons, 1948.

  61. Lyu M R (ed.). Handbook of Software Reliability Engineering. McGraw-Hill, 1996.

  62. Fenton N, Littlewood B, Neil M. Applying Bayesian belief network in systems dependability assessment. In Proc. Safety Critical Systems Symposium, Springer-Verlag, Leeds, UK, 1996, pp. 71–94.

  63. Valiant L G. A theory of the learnable. Communications of the ACM, 1984, 27(11): 1134–1142.

    Article  MATH  Google Scholar 

  64. Angluim D. Queries revisited. Theoretical Computer Science, 2004, 313: 175–194.

    Article  MathSciNet  Google Scholar 

  65. C de la Higuera. A bibliographical study of grammatical inference. Pattern Recognition, 2005, 38: 1332–1348.

    Article  Google Scholar 

  66. Cai K Y, Cangussu J W, DeCarlo R A et al. An overview of software cybernetics. In Proc. the 11th Int. Workshop on Software Technology and Engineering Practice, IEEE Computer Society Press, Amsterdam, Holland, 2003, pp. 77–86.

  67. Belli F, Cai K Y, DeCarlo R A et al. Introduction to the special section on software cybernetics. Journal of Systems and Software, article in press, available online, 2006.

Download references

Author information

Authors and Affiliations

  1. Department of Automatic Control, Beijing University of Aeronautics and Astronautics, Beijing, 100083, P.R. China

    Kai-Yuan Cai

Authors
  1. Kai-Yuan Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai-Yuan Cai.

Additional information

Regular Paper: This work was supported by the National Natural Science Foundation of China (Grant Nos.60233020, 60474006 and 60473067).

Kai-Yuan Cai is a Cheung Kong Scholar (Chair Professor), jointly appointed by the Ministry of Education of China and the Li Ka Shing Foundation of Hong Kong in 1999. He has been a full professor at Beihang University (Beijing University of Aeronautics and Astronautics) since 1995. He was born in April 1965 and entered Beihang University as an undergraduate student in 1980. He received his B.S. degree in 1984, M.S. degree in 1987, and Ph.D. degree in 1991, all from Beihang University. He was a research fellow at the Centre for Software Reliability, City University, London, and a visiting scholar at City University of Hong Kong, Swinburge University of Technology (Australia), University of Technology, Sydney (Australia), and Purdue University (USA). Dr. Cai has published many research papers and is the author of three books: Software Defect and Operational Profile Modeling (Kluwer, Boston, 1998); Introduction to Fuzzy Reliability (Kluwer, Boston, 1996); Elements of Software Reliability Engineering (Tshinghua University Press, Beijing, 1995, in Chinese). His main research interests include software reliability and testing, and intelligent systems and control.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cai, KY. Software Reliability Experimentation and Control. J Comput Sci Technol 21, 697–707 (2006). https://doi.org/10.1007/s11390-006-0697-1

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-006-0697-1

Keywords

Search

Navigation