Skip to main content
Log in

The network of faults: a complex network approach to prioritize test cases for regression testing

  • Original Paper
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

Regression testing is performed to provide confidence that changes in a part of software do not affect other parts of the software. An execution of all existing test cases is the best way to re-establish this confidence. However, regression testing is an expensive process—there might be insufficient resources (e.g., time, workforce) to allow for the re-execution of all test cases. Regression test prioritization techniques attempt to re-order a regression test suite based on some criteria so that highest priority test cases are executed earlier. In this study, we prioritize test cases for regression testing based on the dependency network of faults. In software testing, it is common that some faults are the consequences of other faults (leading faults). Dependent faults can be removed if and only if the leading faults have been removed. Our goal is to prioritize test cases so that test cases that have exposed the leading faults in the system testing phase, are executed first in regression testing. The leading faults are modeled as the most central faults in the fault dependency network. We present ComReg, a test-case prioritization technique based on the dependency network of faults. We model a fault dependency network as a directed graph and identify leading faults to prioritize test cases for regression testing. We use a centrality aggregation technique which considers six network representative centrality metrics to identify leading faults in the fault dependency network. We also discuss the use of fault communities to select an arbitrary percentage of the test cases from a prioritized regression test suite. We conduct a case study that evaluates the effectiveness and applicability of the proposed method. We obtain a fault dependency network from the development of a vocabulary learning software. We found that the fault network is a small-world graph with distinguishable community structure. The leading faults are common in all centralities and a re-ordering of test cases is feasible for regression testing based on those leading faults. Our method outperforms traditional regression testing prioritization techniques in detecting fault dependencies. Our modeling of the network of faults provides insights into the requirement of recognizing fault dependencies while re-ordering regression test suites for both research and practice. The dependency model needs further evaluation and improvement considering relevant resources (e.g., man-hours).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://github.com/ImrulKayes/Tarantula/blob/master/Tarantula1.0.msi.

  2. https://github.com/ImrulKayes/Tarantula.

  3. http://networkx.github.io/.

References

  1. Leung HKN, White L (1989) Insights into regression testing [software testing]. In: Proceedings of conference on software maintenance, pp 60–69

  2. Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Softw Test Verif Reliab 22:67–120

    Article  Google Scholar 

  3. Kayes I (2011) Agile testing: introducing PRAT as a metric of testing quality in scrum. ACM SIGSOFT Softw Eng Notes 36:1–5

    Article  Google Scholar 

  4. Kayes I, Sarker M, Chakareski J (2013) On measuring test quality in scrum: an empirical study. arXiv:1310.2545

  5. Beizer B (2003) Software testing techniques. Dreamtech Press, India

    Google Scholar 

  6. Malishevsky AG, Ruthruff JR, Rothermel G, Elbaum S (2006) Cost-cognizant test case prioritization. Department of Computer Science and Engineering, University of Nebraska-Lincoln, Technical Report

  7. Hamlet D, Maybee J (2000) The engineering of software: a technical guide for the individual, 1st edn. Addison-Wesley Longman Publishing Co., Inc., Boston In:

    Google Scholar 

  8. Elbaum S, Malishevsky A, Rothermel G (2002) Test case prioritization: a family of empirical studies. IEEE Trans Softw Eng 28:159–182

    Article  Google Scholar 

  9. Rothermel G, Untch R, Chu C, Harrold M (1999) Test case prioritization: an empirical study. In: Proceedings of IEEE international conference on software maintenance (ICSM’99), pp 179–188

  10. Rothermel G, Untch R, Chu C, Harrold M (2001) Prioritizing test cases for regression testing. IEEE Trans Softw Eng 27:929–948

    Article  Google Scholar 

  11. Elbaum S, Malishevsky AG, Rothermel G (2000) Prioritizing test cases for regression testing. In: Proceedings of the 2000 ACM SIGSOFT international symposium on software testing and analysis (ISSTA’00). ACM, New York, pp 102–112

  12. Srikanth H, Williams L, Osborne J (2005) System test case prioritization of new and regression test cases. In: Proceedings of international symposium on empirical software engineering

  13. Krishnamoorthi R, Sahaaya Arul Mary S (2009) Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf Softw Technol 51:799–808

    Article  Google Scholar 

  14. Zhang L, Hou SS, Guo C, Xie T, Mei H (2009) Time-aware test-case prioritization using integer linear programming. In: Proceedings of the eighteenth international symposium on software testing and analysis (ISSTA’09). ACM, New York, pp 213–224

  15. Walcott KR, Soffa ML, Kapfhammer GM, Roos RS (2006) Timeaware test suite prioritization. In: Proceedings of the international symposium on software testing and analysis (ISSTA’06). ACM, New York, pp 1–12

  16. Alspaugh S, Walcott KR, Belanich M, Kapfhammer GM, Soffa ML (2007) Efficient time-aware prioritization with knapsack solvers. In: Proceedings of the 1st ACM international workshop on empirical assessment of software engineering languages and technologies (WEASELTech’07): held in conjunction with the 22nd IEEE/ACM international conference on automated software engineering (ASE’07). ACM, New York, pp 13–18

  17. Huang CY, Lin CT (2006) Software reliability analysis by considering fault dependency and debugging time lag. IEEE Trans Reliab 55:436–450

    Article  Google Scholar 

  18. Neumann PG (1990) Cause of AT&T network failure. Risks Dig 9:62

    Google Scholar 

  19. Slabodkin G (1998) Software glitches leave navy smart ship dead in the water. Gov Comput News 13:33727-1

    Google Scholar 

  20. Oppenheimer D, Ganapathi A, Patterson DA (2003) Why do internet services fail, and what can be done about it? In: USENIX symposium on internet technologies and systems, vol 67, Seattle

  21. Kayes M (2011) Test case prioritization for regression testing based on fault dependency. In: Proceedings of 3rd international conference on electronics computer technology (ICECT’11), vol 5, pp 48–52

  22. Rooney P (2002) Microsoft’s CEO: 80-20 rule applies to bugs, not just features. http://www.crn.com/news/security/18821726/microsofts-ceo-80-20-rule-applies-to-bugs-not-just-features.htm. Accessed 25 Nov 2014

  23. Kayes I, Qian X, Skvoretz J, Iamnitchi A (2012) How influential are you: detecting influential bloggers in a blogging community. In: Proceedings of the 4th international conference on social informatics. Springer, Berlin, pp 29–42

  24. Kayes I, Zuo X, Wang D, Chakareski J (2014) To blog or not to blog: characterizing and predicting retention in community blogs. In: Proceedings of the international conference on social computing (SocialCom’14), vol 7. ACM, New York, pp 1–7:8

  25. Kayes I, Zuo X, Wang D, Chakareski J (2014) Did you blog yesterday? Retention in community blogs. In: Proceedings of the international conference on social computing (SocialCom’14). ACM, New York, pp 16:1–16:2

  26. Freeman L (1977) A set of measures of centrality based upon betweenness. Sociometry 40:35–41

    Article  Google Scholar 

  27. Newman MEJ (2010) Networks: an introduction. Oxford University Press, USA

    Book  Google Scholar 

  28. Brandes U (2008) On variants of shortest-path betweenness centrality and their generic computation. Soc Netw 30:136–145

    Article  Google Scholar 

  29. Bonacich P (1972) Factoring and weighting approaches to status scores and clique identification. J Math Sociol 2:113–120

    Article  Google Scholar 

  30. Page L, Brin S, Motwani R, Winograd T (1999) The pagerank citation ranking: bringing order to the web. Stanford InfoLab

  31. Girvan M, Newman ME (2002) Community structure in social and biological networks. Proc Natl Acad Sci 99:7821–7826

    Article  MATH  MathSciNet  Google Scholar 

  32. Tyler JR, Wilkinson DM, Huberman BA (2005) E-mail as spectroscopy: automated discovery of community structure within organizations. Inf Soc 21:143–153

    Article  Google Scholar 

  33. Nazir A, Raza S, Chuah CN (2008) Unveiling facebook: a measurement study of social network based applications. In: Proceedings of the 8th ACM SIGCOMM conference on internet measurement (IMC’08). ACM, New York, pp 43–56

  34. Onnela JP, Saramäki J, Hyvönen J, Szabó G, Lazer D, Kaski K, Kertész J, Barabási AL (2007) Structure and tie strengths in mobile communication networks. Proc Natl Acad Sci 104:7332–7336

    Article  Google Scholar 

  35. Kumar R, Novak J, Raghavan P, Tomkins A (2004) Structure and evolution of blogspace. Commun ACM 47:35–39

    Article  Google Scholar 

  36. Chen J, Yuan B (2006) Detecting functional modules in the yeast protein–protein interaction network. Bioinformatics 22:2283–2290

    Article  Google Scholar 

  37. Newman ME, Girvan M (2004) Finding and evaluating community structure in networks. Phys Rev E 69:026113

    Article  Google Scholar 

  38. Pareto V (1971) Manual of political economy. Macmillan, London

  39. Duong L (2009) Applying the “80–20 rule” with the standish group’s statistics on software usage. http://bit.ly/beBOf1. Accessed 25 Nov 2014

  40. WordHacker (2006) Gre word list. http://www.wordhacker.com/en/article/Barron_gre_list_a.htm. Accessed 25 Nov 2014

  41. Watts DJ, Strogatz S (1998) Collective dynamics of ‘small-world’ networks. Nature 393:440–442

    Article  Google Scholar 

  42. Jeffrey D, Gupta R (2006) Test case prioritization using relevant slices. In: Proceedings of 30th annual international computer software and applications conference (COMPSAC’06), vol 1, pp 411–420

  43. Zhi-hua Z, Yong-min M, Ying-ai T (2012) Test case prioritization for regression testing based on function call path. In: Fourth international conference on computational and information sciences (ICCIS), pp 1372–1375

  44. Blondel VD, Guillaume JL, Lambiotte R, Lefebvre E (2008) Fast unfolding of communities in large networks. J Stat Mech Theory Exp 2008:P10008

    Article  Google Scholar 

  45. Do H, Rothermel G, Kinneer A (2004) Empirical studies of test case prioritization in a JUnit testing environment. In: 15th international symposium on software reliability engineering (ISSRE’04), pp 113–124

  46. Korel B, Tahat LH, Harman M (2005) Test prioritization using system models. In: Proceedings of the 21st IEEE international conference on software maintenance (ICSM’05). IEEE, pp 559–568

  47. Korel B, Koutsogiannakis G, Tahat L (2008) Application of system models in regression test suite prioritization. In: IEEE international conference on software maintenance (ICSM’08), pp 247–256

  48. Jones J, Harrold M (2003) Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans Softw Eng 29:195–209

    Article  Google Scholar 

  49. Kim JM, Porter A (2002) A history-based test prioritization technique for regression testing in resource constrained environments. In: Proceedings of the 24th international conference on software engineering (ICSE’02), pp 119–129

  50. Sherriff M, Lake M, Williams L (2007) Prioritization of regression tests using singular value decomposition with empirical change records. In: The 18th IEEE international symposium on software reliability (ISSRE’07), pp 81–90

  51. Leon D, Podgurski A (2003) A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases. In: Proceedings of 14th international symposium on software reliability engineering (ISSRE’03), pp 442–453

  52. Sampath S, Bryce R, Viswanath G, Kandimalla V, Koru A (2008) Prioritizing user-session-based test cases for web applications testing. In: Proceedings of 1st international conference on software testing, verification, and validation, pp141–150

  53. Rummel MJ, Kapfhammer GM, Thall A (2005) Towards the prioritization of regression test suites with data flow information. In: Proceedings of the ACM symposium on applied computing (SAC’05). ACM, New York, pp 1499–1504

  54. Qu B, Nie C, Xu B, Zhang X (2007) Test case prioritization for black box testing. In: 31st annual international of computer software and applications conference (COMPSAC’07), vol 1, pp 465–474

  55. Faloutsos M, Faloutsos P, Faloutsos C (1999) On power-law relationships of the internet topology. SIGCOMM Comput. Commun. Rev. 29:251–262

    Article  Google Scholar 

  56. Aiello W, Chung F, Lu L (2001) Random evolution in massive graphs. In: Proceedings of 42nd IEEE symposium on foundations of computer science, pp 510–519

  57. Redner S (1998) How popular is your paper? An empirical study of the citation distribution. Eur Phys J B-Condens Matter Complex Syst 4:131–134

    Article  Google Scholar 

  58. Kayes I, Kourtellis N, Quercia D, Iamnitchi A, Bonchi F (2015) The social world of content abusers in community question answering. In: Proceedings of the 24th international conference on world wide web (WWW’15). International world wide web conferences steering committee, Republic and Canton of Geneva, pp 570–580

  59. Newman ME, Strogatz SH, Watts DJ (2001) Random graphs with arbitrary degree distributions and their applications. Phys Rev E 64:026118

    Article  Google Scholar 

  60. Jiang K, Ediger D, Bader D (2009) Generalizing \(k\)-betweenness centrality using short paths and a parallel multithreaded implementation. In: international conference on parallel processing (ICPP’09), pp 542–549

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Imrul Kayes.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kayes, I., Islam, S. & Chakareski, J. The network of faults: a complex network approach to prioritize test cases for regression testing. Innovations Syst Softw Eng 11, 261–275 (2015). https://doi.org/10.1007/s11334-015-0255-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11334-015-0255-5

Keywords

Navigation