Modeling and analysis of reliability of multi-release open source software incorporating both fault detection and correction processes

https://doi.org/10.1016/j.jss.2016.01.025Get rights and content

Highlights

  • The failure process in testing multi-release software is analyzed with the consideration of faults correction delay.

  • Two kinds of multi-release software reliability model are proposed.

  • The model is validated on real test datasets from open source software.

  • A comprehensive analysis of optimal release times based on cost-efficiency is provided.

Abstract

Large software systems require regular upgrading that tries to correct the reported faults in previous versions and add some functions to meet new requirements. It is thus necessary to investigate changes in reliability in the face of ongoing releases. However, the current modeling frameworks mostly rely on the idealized assumption that all faults will be removed instantaneously and perfectly. In this paper, the failure processes in testing multi-release software are investigated by taking into consideration the delays in fault repair time based on a proposed time delay model. The model is validated on real test datasets from the software that has been released three times with new features. A comprehensive analysis of optimal release times based on cost-efficiency is also provided, which could help project managers to determine the best time to release the software.

Introduction

The software industry is growing rapidly and has become very competitive. As a result, many software developers are cutting back their schedules to ensure prompt delivery and developing new features to keep their products competitive. This is especially true for large and complex software. After a release, reported faults in previous versions will be removed and new functions may be designed to meet new requirements in new versions. Developers generally pay greater attention to balancing competition in the market, and thus risk quality because of the short software lifecycle. The upgradation process constitutes a challenge for software companies looking to produce highly reliable software and ensure the release time is on schedule.

Over the past four decades, researchers have studied a variety of methods to assess software reliability. One of the most widely investigated and applied of those methods is the software reliability growth model (SRGM) (Lyu, 2007, Amin et al., 2013, Febrero et al., 2014, Yamada, 2014). Most SRGMs utilize the fault data collected during the test process to describe the stochastic behavior of the software fault detection process (FDP) with respect to time, and it is reasonable to assume that the fault counts in each time interval are mutually independent of each other (Amin et al., 2013). Non-homogeneous Poisson process (NHPP) model is considered as one of the most effective models (Goel and Okumoto, 1979, Lyu, 1996, Ohishi et al., 2009). They have been successfully applied in many software projects to manage tests and predict operational reliability (Jeske and Zhang, 2007, Lin and Huang, 2008, Rana et al., 2014). They have also been utilized in making critical decisions, such as those involved in cost-benefit analysis, resource allocation, and release-time determination (Peng et al., 2013, Park and Baik, 2015, Wang et al., 2015).

Furthermore, a number of specific SRGMs have been proposed for investigating the reliability of Open Source software (OSS), which is a growing area of software development and applications. For example, Tamura and Yamada (2013) propose a method of software reliability assessment for the embedded OSS with flexible hazard rate modeling. Pachauri et al. (2013) blended fuzzy set theory with software reliability measurement and total cost analysis, and Gratus and Pratibha (2013) proposed an approach for carrying out pre-statistical data analyses based on assessment of software's reliability metrics. Luan and Huang (2014) proposed an improved Pareto distribution model for analyzing the failure process, although their method is confined to ungrouped data.

In this paper, the failure process in testing multi-release software is further explored by taking into consideration a delay in the fault repair time based on the time-delay model proposed by Wu et al. (2007). Both fault-correction and detection processes are considered. It is assumed that the faults in a new version comprise both undetected faults in a previous version and new faults introduced during the development process of the new version. A framework for assessing the expected number of remaining faults in each version is proposed and the optimal release time for each version is also investigated.

The remainder of this paper is organized as follows. Section 2 outlines the proposed framework for multi-release software modeling. In Section 3, the parameter estimation with Least Square Estimation (LSE) is developed and the optimal release strategy for such software is discussed. Section 5 demonstrates the application of the proposed models with a three-release dataset collected from a practical OSS test process and presents the results of optimal release time analysis. Finally, the conclusion is given in Section 6.

Section snippets

Modeling the multi-release situation

Most of the existing SRGMs focus on the software development process of only a single version. It is thus necessary to investigate changes in reliability arising from ongoing releases, which is a rather complex problem as usually there are many reasons for a new release. Several studies have been carried out in this regard in the literature. For example, Smidts et al. (1998) applied software failure data from a previous release to perform reliability estimation on a current release, and

Single-release modeling framework for FDP and FCP

For single-version software, the method of modeling FDP is like the traditional NHPP SRGM in which the cumulative number of detected faults, N(t), is assumed to follow a Poisson distribution with mean value function (MVF) md(t), i.e., P{N(t)=n}=mdn(t)n!emd(t).

According to the basic assumption of fault removal, the MVF can be given by {dmd(t)dt=λd(t)=F(t)1F(t)[amd(t)]md(0)=0,where λd(t) refers to the failure rate during the test process and F(t) is a cumulative distribution function. In

Parameter estimation

Nonlinear LSE is applied to estimate the model parameters, and those estimates are the optimal solution to the following nonlinear programming problem: minθS(θ)=j=1n[i=1kj(mdj(ti,θ)nji)2+i=1kj(mcj(ti,θ)mji)2],where nij and mij refer to the observed cumulative fault counts in the FDP and FCP of version j by testing time ti, and θ constitutes the model parameters. The foregoing extreme problem can be solved with the build-in function in MATLAB.

Optimal upgrading planning problem for multi-release software

When a company is planning to develop

Numerical illustration

In this section, two numerical application examples are given for the illustration purpose. The applied datasets are from online bug tracking systems (Mozilla and Gnome). Many organizations, particularly enterprises using OSS, utilize such systems in managing their software development processes.

In the numerical example, only those faults that are not duplicate and are reproducible for others are selected. The detection time of each fault is the opening time of this fault record, and the

Discussion

In this paper, a framework for modeling multi-release software reliability is first developed and parameter estimation problem in this situation is also studied. The developed approach takes into consideration delays in fault repair time based on the time delay model. The research incorporates software upgrading that has been an efficient approach to dealing with the increasing competition in today's market. It is assumed that the faults in a new software version comprise both undetected faults

Acknowledgments

The authors sincerely thank the editor and the reviewers of an earlier version of this paper. The comments have led to significant improvement in the presentation and additional research carried out.

Jianfeng Yang is an Associate Professor in Guizhou University. He received his Ph.D. degree in software engineering and reliability analysis in Guizhou University. His research interests include reliability engineering and web system reliability and availability, and applied statistics.

References (36)

  • PachauriB. et al.

    Incorporating inflection S-shaped fault reduction factor to enhance software reliability growth

    Appl. Math. Model.

    (2015)
  • PachauriB. et al.

    Modeling optimal release policy under fuzzy paradigm in imperfect debugging environment

    Inf. Softw. Technol.

    (2013)
  • ParkJ. et al.

    Improving software reliability prediction through multi-criteria based dynamic model selection and combination

    J. Syst. Softw.

    (2015)
  • PengR. et al.

    Testing effort dependent software reliability model for imperfect debugging process considering both detection and correction

    Reliab. Eng. Syst. Saf.

    (2014)
  • RanaR. et al.

    Selecting software reliability growth models and improving their predictive accuracy using historical projects data

    J. Syst. Softw.

    (2014)
  • WangJ. et al.

    An imperfect software debugging model considering log-logistic distribution fault content function

    J. Syst. Softw.

    (2015)
  • DaiY. et al.

    Uncertainty analysis in software reliability modeling by Bayesian approach with maximum-entropy principle

    IEEE Trans. Softw Eng.

    (2007)
  • GarmabakiA.H.S. et al.

    Multi up-gradation software reliability growth model with faults of different severity

  • Cited by (77)

    View all citing articles on Scopus

    Jianfeng Yang is an Associate Professor in Guizhou University. He received his Ph.D. degree in software engineering and reliability analysis in Guizhou University. His research interests include reliability engineering and web system reliability and availability, and applied statistics.

    Yu Liu is a Ph.D. student in the department of Systems Engineering and Engineering Management, City University of Hong Kong. His research interests include software reliability growth model, reliability prediction for software/hardware system. Liu received a MS in nuclear engineering from Tsinghua University.

    Min Xie is an elected fellow of IEEE and a Chair Professor of Industrial Engineering at City University of Hong Kong. He received his PhD in Quality Technology in 1987 from Linkoping University in Sweden. He was awarded the prestigious LKY research fellowship in 1991 and he is a Professor at National University of Singapore.

    Ming Zhao is a Ph.D. supervisor at Guizhou University. He received his B.S. degree in mathematics from Guizhou University in 1982, M.S. degree in statistics from Wuhan University, China, in 1986, and Ph.D. degree in industrial engineering from Linkoping University, Sweden, 1994. Before he became a Ph.D. candidate at Linkoping University, he had served three years as a reliability engineer at China Academy of Launch Vehicle Technology, Beijing, China. In 1997, he joined the Department of Technology, University of Gavle, Sweden. Since 2002, his research interests include reliability engineering, software quality engineering, and applied statistics.

    View full text