Skip to main content
SpringerLink
Log in

Meta-learning based selection of software reliability models

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

The correct estimation of the software reliability level is fundamental to reduce efforts in the testing, maintenance and release activities. To help in this task, we find in the literature an increasing number of software reliability models (SRMs). However, none has proved to perform well considering different contexts. Due to this, the selection of the best model for a particular case is an important task. Most existing works on SRM selection need to test different models and decide based on how well the model fits the data and predicts the future events. Moreover, in general, they do not consider search-based models. Considering this fact, this paper introduces a Meta-learning approach for SRM selection. In such approach, some meta-features are used to indicate the best performing model. The approach is independent of the type of models to be selected, and can be used with different data mining algorithms. It includes the following activities: meta-knowledge extraction, meta-learning and classification. The activities meta-knowledge extraction and meta-learning are performed just once and generate a meta-classifier. Therefore, the meta-classifier is used to select the most adequate model for new projects (classification activity). The approach is evaluated in a set of experiments and the results do not show statistical difference between the Meta-learning approach and the choice of the best performing model. Otherwise, the results point out statistical difference between the Meta-learning approach and the choice of the worst performing model with a large stochastic difference according to the Vargha and Delaney Effect Size.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. The data from the work of Costa et al. (2007) used in our experiments are available as supplementary material and can be found at http://www.inf.ufpr.br/gres/apoio_en.html.

References

  • Afzal, W., Torkar, R.: Suitability of genetic programming for software reliability growth modeling. In: International Symposium on Computer Science and its Applications, pp. 114–117 (2008)

  • Aljahdali, S.H., Sheta, A., Rine, D.: Prediction of software reliability: a comparison between regression and neural network non-parametric models. In: ACS/IEEE International Conference on Computer Systems and Aplications, pp. 470–473 (2001)

  • Arlot, S., Celisse, A.: A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40–79 (2010). doi:10.1214/09-SS054

    Article  MathSciNet  MATH  Google Scholar 

  • Asad, C.A., Ullah, M.I., Rehman, M.J.: An approach for software reliability model selection. In: 28th Annual International Computer Software and Applications Conference (COMPSAC’04), pp. 534–539. IEEE Computer Society (2004)

  • Borowska, K., Topczewska, M.: Data preprocessing in the classification of the imbalanced data. Adv. Comput. Sci. Res. 11, 31–46 (2014)

    Google Scholar 

  • Brazdil, P., Giraud-Carrier, C., Soares, C., Vilalta, R.: Metalearning: Applications to Data Mining (Cognitive Technologies). Springer, Berlin (2009)

    MATH  Google Scholar 

  • Caiuta, R., Emmendorfer, L., Pozo, A., Vergilio, S.R.: Selecting software reliability models with a neural network classifier. In: IEEE International Joint Conference on Neural Networks (IJCNN) (2008)

  • Chatzis, S., Andreou, A.: Maximum entropy discrimination poisson regression for software reliability modeling. Neural Netw. Learn. Syst. PP(99), 1–1 (2015)

    Google Scholar 

  • Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    MATH  Google Scholar 

  • Costa, E.O., Souza, G.A., Pozo, A., Vergilio, S.R.: Exploring genetic programming and boosting techiques to model software reliability. IEEE Trans. Reliab. 56(3), 422–434 (2007)

    Article  Google Scholar 

  • Demsar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)

    MathSciNet  MATH  Google Scholar 

  • Febrero, F., Calero, C., ngeles Moraga, M.: A systematic mapping study of software reliability modeling. Inf. Softw. Technol. 56(8), 839–849 (2014)

    Article  Google Scholar 

  • Haykin, S.: Neural Networks: A Comprehensive Foundation, 2nd edn. Prentice Hall, Upper Saddle River (1998)

    MATH  Google Scholar 

  • Kartalopoulos, S.: Understanding Neural Networks and Fuzzy Logic: Basic Concepts and Applications. Wiley, New York (1996)

    MATH  Google Scholar 

  • Karunanithi, N., Whitley, D., Malaiya, Y.K.: Prediction of software reliability using connectionist models. IEEE Trans. Softw. Eng. 18(7), 563–574 (1992)

    Article  Google Scholar 

  • Khoshgoftaar, T., Woodcock, T.: Software reliability model selection: a case study. In: IEEE International Symposium on Software Reliability Engineering, pp. 183–191 (1991)

  • Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International Conference on Artificial Intelligence (1995)

  • Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47(260), 583–621 (1952)

    Article  MATH  Google Scholar 

  • Lakey, P., Neufelder, A.: System and Software Reliability Assurance Notebook. Rome Laboratory, Rome (1997)

    Google Scholar 

  • Lyu, M.: Handbook of software reliability engineering. IEEE Computer Society Press and McGraw-Hill. http://www.cse.cuhk.edu.hk/~lyu/book/reliability/ (2005). Accessed Jan 2008

  • Manderna, R.: Software reliability model selection criteria: a literature review. Softw. Eng. Technol. 5(6), 200–204 (2013)

    Google Scholar 

  • Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)

    MATH  Google Scholar 

  • Moranda, P.: Predictions of software reliability during debugging. In: Proceedings of the Annual Reliability Maintainability Symposium (1975)

  • Moranda, P., Jelinski, Z.: Final report on software reliability study. Tech. rep, McDonnall Douglas Astronautics Company (1972)

  • Musa, J.: A theory of software reliability and its application. IEEE Trans. Softw. Eng. 1(3), 312–327 (1975)

    Article  Google Scholar 

  • Musa, J.: Software Reliability Data. Data & Analysis Center for Software, New York (1980)

    Google Scholar 

  • Musa, J.D., Ianino, A., Okumoto, K.: Software Reliability: Measurement, Prediction, Application. McGraw-Hill, New York (1990)

    Google Scholar 

  • Pai, G.: A survey of software reliability models. Tech. rep., DCE, University of Virginia (2002)

  • Pappa, G., Ochoa, G., Hyde, M., Freitas, A., Woodward, J., Swan, J.: Contrasting meta-learning and hyper-heuristic research: the role of evolutionary algorithms. Genet. Program. Evol. Mach. 15(1), 3–35 (2014)

    Article  Google Scholar 

  • Park, J., Baik, J.: Improving software reliability prediction through multi-criteria based dynamic model selection and combination. J. Syst. Softw. 101, 236–244 (2015)

    Article  Google Scholar 

  • Prudencio, R., Ludermir, T.: Meta-learning approaches to selecting time series models. Neurocomput. J. 61, 121–137 (2004)

    Article  Google Scholar 

  • Prudencio, R., Ludermir, T., Carvalho, F.: A modal symbolic classifier to select time series models. Pattern Recogn. Lett. 25(8), 911–921 (2004)

    Article  Google Scholar 

  • Quinlan, J.R.: C4.5 Programs for Machine Learning. Morgan Kaufmann Publishers, San Francisco (1993)

    Google Scholar 

  • Santos, P., Ludemir, T., Prudencio, R.: Selection of time series forecasting models based on performance information. In: Proceedings of Fourth International Conference on Hybrid Intelligent Systems (HIS 04) (2004)

  • Siegal, S., Castellan, N.: Non-Parametric Statistics for the Behavioural Sciences. McGraw Hill, New York (1988)

    Google Scholar 

  • Sitte, R.: Comparison of software reliability growth predictions: neural networks vs parametric recalibration. IEEE Trans. Softw. Eng. 48(3), 285–291 (1999)

    Google Scholar 

  • Smith-Miles, K.A.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41, 0300–0360 (2008)

    Article  Google Scholar 

  • Souza, G.A., Vergilio, S.R.: Modeling software reliability growth with artificial neural networks. In: IEEE Latin American Test Workshop, pp. 165–170. Buenos Aires (2006)

  • Sundarkumar, G.G., Ravi, V.: A novel hybrid undersampling method for mining unbalanced datasets in banking and insurance. Eng. Appl. Artif. Intell. 37, 368–377 (2015)

    Article  Google Scholar 

  • University, W.: Weka: Machine Lerning Software in Java. University of Waikato. http://www.cs.waikato.ac.nz/ml/weka (2007)

  • Vargha, A., Delaney, H.D.: A critique and improvement of the cl common language effect size statistics of mcgraw and wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)

    Google Scholar 

  • Vilalta, R., Giraud-Carrier, C., Brazdil, P.: Meta-learning: concepts and techniques. In: Maimon, O., Rokach, L. (eds.) Data mining and knowledge discovery handbook (2005)

  • Wu, Y., Yang, R.: Software reliability modeling based on svm and virtual sample. In: Reliability and Maintainability Symposium (RAMS), 2013 Proceedings-Annual, pp. 1–6 (2013)

  • Yamada, S.: Software Reliability Modeling: Fundamentals and Applications. Springer, Berlin (2014)

    Book  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank CNPq for financial support.

Author information

Authors and Affiliations

  1. Computer Science Department, Federal University of Paraná (UFPR), CP 19:081, Curitiba, CEP: 81531-970, Brazil

    Rafael Caiuta, Aurora Pozo & Silvia Regina Vergilio

Authors
  1. Rafael Caiuta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aurora Pozo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Silvia Regina Vergilio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Silvia Regina Vergilio.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Caiuta, R., Pozo, A. & Vergilio, S.R. Meta-learning based selection of software reliability models. Autom Softw Eng 24, 575–602 (2017). https://doi.org/10.1007/s10515-016-0195-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-016-0195-9

Keywords

Search

Navigation