skip to main content
research-article

Test Case Prioritization Using Extended Digraphs

Published:02 December 2015Publication History
Skip Abstract Section

Abstract

Although many test case prioritization techniques exist, their performance is far from perfect. Hence, we propose a new fault-based test case prioritization technique to promote fault-revealing test cases in model-based testing (MBT) procedures. We seek to improve the fault detection rate—a measure of how fast a test suite is able to detect faults during testing—in scenarios such as regression testing. We propose an extended digraph model as the basis of this new technique. The model is realized using a novel reinforcement-learning (RL)- and hidden-Markov-model (HMM)-based technique which is able to prioritize test cases for regression testing objectives. We present a method to initialize and train an HMM based upon RL concepts applied to an application's digraph model. The model prioritizes test cases based upon forward probabilities, a new test case prioritization approach. In addition, we also propose an alternative approach to prioritizing test cases according to the amount of change they cause in applications. To evaluate the effectiveness of the proposed techniques, we perform experiments on graphical user interface (GUI)-based applications and compare the results with state-of-the-art test case prioritization approaches. The experimental results show that the proposed technique is able to detect faults early within test runs.

References

  1. K. Aas and L. Eikvil. 1996. Text page recognition using grey-level features and hidden Markov models. Patt. Recogn. 29, 6, 977--985.Google ScholarGoogle ScholarCross RefCross Ref
  2. A. Arcuri and L. Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (ICSE'11). IEEE, 1--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. I. Arel, C. Liu, T. Urbanik, and A. G. Kohls. 2010. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intell. Transport Syst. 4, 2, 128--135.Google ScholarGoogle ScholarCross RefCross Ref
  4. D. Ariu, R. Tronci, and G. Giacinto. 2011. HMMPayl: An intrusion detection system based on hidden Markov models. Comput. Secur. 30, 4, 221--241. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. S. Arlt, S. Pahl, C. Berolini, and M. Schaf. 2012. Trends in model-based GUI testing. Adv. Comput. 86, 183--222.Google ScholarGoogle ScholarCross RefCross Ref
  6. A. Avritzer and E. R. Weyuker. 1995. The automatic generation of load test suites and the assessment of the resulting software. IEEE Trans. Softw. Engin. 21, 9, 705--716. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. C. G. Bai, Q. P. Hu, M. Xie, and S. H. Ng. 2005. Software failure prediction based on a Markov Bayesian network model. J. Syst. Softw. 74, 3, 275--282. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. L. Baum, T. Petrie, G. Soules, and N. Weiss. 1970. A maximization technique occuring in the statistical analysis of probabilistic functions of Markov chains. The Ann. Math. Statist. 41, 1, 164--171.Google ScholarGoogle ScholarCross RefCross Ref
  9. G. Becce, L. Mariani, O. Riganelli, and M. Santoro. 2012. Extracting widget descriptions from GUIs. In Proceedings of the 15th International Conference on Fundamental Approaches to Software Engineering (FASE'12). 347--361. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. P. Blunsom. 2004. Hidden Markov models. http://digital.cs.usu.edu/∼cyan/CS7960/hmm-tutorial.pdf.Google ScholarGoogle Scholar
  11. K.-Y. Cai, Y.-C. Li, and W.-Y. Ning. 2005. Optimal software testing in the setting of controlled Markov chains. Euro. J. Oper. Res. 162, 2, 552--579.Google ScholarGoogle ScholarCross RefCross Ref
  12. H. S. Chang. 2006. Reinforcement learning with supervision by combining multiple learnings and expert advices. In Proceedings of the American Control Conference (ACC'06). 159--161.Google ScholarGoogle Scholar
  13. W. Ching. 2006. Markov Chains: Model, Algorithms and Applications. Springer Science. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. J. Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associate Publishers.Google ScholarGoogle Scholar
  15. H. Cua and N. Dethlefs. 2011. Hierarchical reinforcement learning and hidden Markov models for task-oriented natural language generation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short Papers (ACL'11). 654--659. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. R. Davoodi and B. J. Andrews. 1998. Computer simulation of FES standing up in paraplegia: A self-adaptive fuzzy controller with reinforcement learning. IEEE Trans. Rehabil. Engin. 6, 2, 151--161.Google ScholarGoogle ScholarCross RefCross Ref
  17. P. Dayan and C. J. Watkin. 1992. Technical note Q-learning. Mach. Learn. 8, 279--292. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. R. Dev, A. Jaaskelainen, and M. Katara. 2012. Model-based GUI testing: Case smartphone camera and messaging development. Adv. Comput. 85, 65--122.Google ScholarGoogle ScholarCross RefCross Ref
  19. H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel. 2010. The effect of time constraints in test case prioritization: A series of controlled experiments. IEEE Trans. Softw. Engin. 36, 5, 592--617. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. I. K. El-Far and J. A. Whittaker. 2002. Model-based software testing. In Encyclopedia of Software Engineering. Wiley, 1--22.Google ScholarGoogle Scholar
  21. S. Elbaum, G. Rothermel, A. G. Malishevsky, and S. Member. 2002. Test case prioritization: A family of empirical studies test case prioritization. IEEE Trans. Softw. Engin. 27, 10, 929--948.Google ScholarGoogle Scholar
  22. D. P. Ellis. 2010. The Essential Guide to Effect Sizes; Statistical Power, Meta-Analysis and the Interpretation of Research Results. Cambridge University Press.Google ScholarGoogle Scholar
  23. A. T. Endo and A. Simao. 2011. Model-based testing of service-oriented applications via state models. In Proceedings of the IEEE International Conference on Services Computing (SCC'11). 432--439. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. U. Farooq. 2011. Model based test suite minimization using metaheuristics. http://ro.ecu.edu.au/cgi/viewcontent.cgi?articlen=1409&context==theses.Google ScholarGoogle Scholar
  25. A. Feliachi and H. L. Guen. 2010. Generating transition probabilities for automatic model-based test generation. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation (ICST'10). 99--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. P. G. Frankl, G. Rothermel, K. Sayre, and F. I. Vokolos. 2003. An empirical comparison of two safe regression test selection techniques. In Proceedings of the International Symposium on Empirical Software Engineering (ISESE'03). IEEE Computer Society, 195--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. F. Galon. 1889. Natural Inheritance. MacMillan.Google ScholarGoogle Scholar
  28. K. Hamamoto, K. Morooka, and H. Nagahashi. 2004. Motion recognition by combining HMM and reinforcement learning. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (ICSMC'04). 259--264.Google ScholarGoogle Scholar
  29. J. Hao and E. Mendes. 2006. Usage-based statistical testing of Web applications. In Proceedings of the 6th International Conference on Web Engineering (ICWE'06). ACM Press, New York, 17--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. H. Hemmati. 2011. Similarity-based test case selection: Toward scalable and practical model-based testing. https://www.simula.no/publications/similarity-based-test-case-selection-toward-scalable-and-practical-model-based-testing.Google ScholarGoogle Scholar
  31. H. Hemmati, A. Arcuri, and L. Briand. 2013. Achieving scalable model-based testing through test case diversity. ACM Trans. Softw. Engin. Methodol. 22, 1, 1--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. M. A. T. Ho, Y. Yamada, and Y. Umetani. 2002. An HMM-based temporal difference learning with model-updating capability for visual tracking of human communicational behaviors. In Proceedings of the 5th IEEE International Conference on Automatic Face Gesture Recognition (FGR'02). 170--175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. H. Hsu and P. A. Lachenbruch. 2008. Paired t-test. In Encyclopedia of Clinical Trials. Wiley, 1--3.Google ScholarGoogle Scholar
  34. C.-Y. Huang, J.-R. Chang, and Y.-H. Chang. 2010. Design and analysis of GUI test-case prioritization using weight-based methods. J. Syst. Softw. 83, 4, 646--659. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. R. Ihaka and R. Gentleman. 1996. R: A language for data analysis and graphics. J. Comput. Graph. Statist. 5, 3, 299--314.Google ScholarGoogle ScholarCross RefCross Ref
  36. T. P. Jaakkola, S. P. Singh, and M. I. Jordan. 1995. Reinforcement learning algorithm for partially observable Markov decision problems. http://papers.nips.cc/paper/951-reinforcement-learning-algorithm-for-partially-observable-markov-decision-problems.pdf.Google ScholarGoogle Scholar
  37. D. Jeffrey and N. Gupta. 2007. Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Trans. Softw. Engin. 33, 2, 108-123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J. Kabudian, M. R. Meybodi, and M. M. Homayounpour. 2004. Applying continuous action reinforcement learning automata (CARLA) to global training of hidden Markov models. In Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'04). 638-642. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. K. Lee, H. Hon, M. Hwang, and X. Huang. 1990. Speech recognition using hidden Markov model: A CMU perspective. Speech Comm. 9, 5--6, 497--508. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. B. Legeard and M. Utting. 2006. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, San Fransisco. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. L. J. Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn. 8, 3--4, 293--321. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. F. Lindlar, A. Windisch, and J. Wegener. 2010. Integrating model-based testing with evolutionary functional testing. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW'10). 163--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. L. Mariani, M. Pezz, O. Riganelli, and M. Santoro. 2011. AutoBlackTest: A tool for automatic black-box testing. In Proceedings of the 33rd International Conference on Software Engineering (ICSE'11). 1013--1015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. L. Mariani, M. Pezz, O. Riganelli, and M. Santoro. 2012. AutoBlackTest: Automatic black-box testing of interactive applications. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST'12). 81--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. A. K. McCallum. 1995. Reinforcement learning with selective percemption and hidden state. Doctoral dissertation, University of Rochester, NY. http://web.media.mit.edu/∼tristan/Classes/MAS.945/Papers/Contextual/McCallum_Thesis.pdf. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. M. McPartland and M. Gallagher. 2011. Reinforcement learning in first person shooter games. IEEE Trans. Comput. Intell. AI Games 3, 1, 43--56.Google ScholarGoogle ScholarCross RefCross Ref
  47. H. Mei, D. Hao, L. Zhang, J. Zhou, and G. Rothermel. 2012. A static approach to prioritizing J-unit test cases. IEEE Trans. Softw. Engin. 38, 6, 1258--1275. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. A. M. Memon. 2007. An event-flow model of GUI-based applications for testing. Softw. Test. Verif. Reliab. 17, 3, 137--157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. A. M. Memon and D. R. Hackner. 2008. Test case generator for GUITAR. In Proceedings of the Companion to the International Conference of Software Engineering (ICSE'08). 959--960. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. S. Mirarab and L. Tahvildari. 2007. A prioritization approach for software. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE'07). 276--290. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. S. Mirarab and L. Tahvildari. 2008. An empirical study on Bayesian network-based approach for test case prioritization. In Proceedings of the 1st International Conference on Software Testing, Verification, and Validation (ICST'08). 278--287. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. D. Nardo, N. Alshahwan, and L. Briand. 2013. Coverage-based test case prioritisation: An industrial case study. In Proceeding of the 6th International Conference on Software Testing, Verification, and Validation (ICST'13). 302--311. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. S. Parsa and A. Khalilian. 2010. On the optimization approach towards test suite minimization approach. Int. J. Softw. Engin. Appl. 4, 1, 15--28.Google ScholarGoogle Scholar
  54. L. R. Rabiner and B. H. Juang. 1986. An introduction to hidden Markov models. IEEE ASSP Mag. 3, 1, 4--16.Google ScholarGoogle ScholarCross RefCross Ref
  55. H. Reza, S. Endapally, and E. Grant. 2007. A model-based approach for testing GUI using hierarchical predicate transition nets. In Proceedings of the 4th International Conference on Information Technology (ITNG'07). 366--370. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. J. A. Rice. 1994. Mathematical Statistics and Data Analysis. Duxbury Press.Google ScholarGoogle Scholar
  57. M. Robinson and P. A. Vorobiev. 1999. Swing: A Fast-Paced Guide with Production-Quality Code Examples. Manning. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. G. Rothermel and D. Hall. 1997. A safe, efficient regression test selection technique. ACM Trans. Softw. Engin. Methodol. 6, 2, 173--210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. G. Rothermel and M. J. Harrold. 1998. Empirical studies of a safe regression test selection technique. IEEE Trans. Softw. Engin. 24, 6, 108--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the International Conference on Software Maintenance (ICSM'98). 34--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. G. Rothermel, R. Untch, and M. J. Harrold. 2001. Prioritizing test cases for regression testing. IEEE Trans. Softw. Engin. 27, 10, 929--948. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. G. S. Semmel and D. G. Linton. 1998. Determining optimal testing times for Markov chain usage models. In Proceedings of the IEEE SECON Conference (Southeastcon'98). 1--4.Google ScholarGoogle Scholar
  63. R. S. Sutton and A. G. Barto. 1998. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, Massachusetts. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. C. Szepesvári. 2010. Algorithms for Reinforcement Learning. Morgan and Claypool. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein. 2012. Static test case prioritization using topic models. Empir. Softw. Engin. 19, 1, 182--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. O. Tramasco and S. Bauer. 2013. Package ‘RHmm’. http://www2.uaem.mx/r-mirror/web/packages/RHmm/RHmm.pdf.Google ScholarGoogle Scholar
  67. M. A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. J. Artif. Intell. Res. 12, 387--416. Google ScholarGoogle ScholarCross RefCross Ref
  68. C. J. Watkin. 1989. Learning from delayed rewards. 1--14. https://www.cs.rhul.ac.uk/home/chrisw/new_thesis.pdf.Google ScholarGoogle Scholar
  69. A. L. White and J. A. Sjorgen. 1988. Markov chains for testing redundant software. In Proceedings of the Reliability and Maintainability Symposium (RAMS'88). 426--433.Google ScholarGoogle Scholar
  70. J. A. Whittaker and M. G. Thomason. 1994. A Markov chain model for statistical software testing. IEEE Trans. Softw. Engin. 20, 10, 812--824. Google ScholarGoogle ScholarDigital LibraryDigital Library
  71. M. Wiering and M. Van Otterlo. 2012. Reinforcement Learning (State of the Art). Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. J. D. Williams and S. Young. 2007. Partially observable Markov decision processes for spoken dialog systems. Comput. Speech Lang. 21, 2, 393--422. Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. 1997. A study of effective regression testing in practice. In Proceedings of the 8th IEEE International Symposium on Software Reliability Engineering (ISSRE'97). 264--274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. M. M. Yin and J. T. L. Wang. 2001. Effective hidden Markov models for detecting splicing junction sites in DNA sequences. Inf. Sci. 139, 1--2, 139--163. Google ScholarGoogle ScholarDigital LibraryDigital Library
  75. S. Yoo and M. Harman. 2007. Regression testing minimisation, selection and prioritisation: A survey. Softw. Test. Verif. Reliab. 7, 1--60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  76. Y. T. Yu and M. F. Lau. 2012. Fault-based test suite prioritization for specification-based testing. Inf. Softw. Technol. 54, 2, 179--202. Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei. 2009a. Time-aware test-case prioritization using integer linear programming. In Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA'09). ACM Press, New York, 401--419. Google ScholarGoogle ScholarDigital LibraryDigital Library
  78. Z. Zhang, W. K. Chan, and T. H. Tse. 2009b. Adaptive random test case prioritization. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE'09). Google ScholarGoogle ScholarDigital LibraryDigital Library
  79. F. Zhen And P. Chenglian. 2004. A system test methodology based on the Markov chain usage model. In Proceedings of the 8th International Conference on Computer Supported Cooperative Work in Design (CACWD'04). 160--165.Google ScholarGoogle Scholar

Index Terms

  1. Test Case Prioritization Using Extended Digraphs

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Software Engineering and Methodology
        ACM Transactions on Software Engineering and Methodology  Volume 25, Issue 1
        December 2015
        339 pages
        ISSN:1049-331X
        EISSN:1557-7392
        DOI:10.1145/2852270
        Issue’s Table of Contents

        Copyright © 2015 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 2 December 2015
        • Accepted: 1 June 2015
        • Revised: 1 November 2014
        • Received: 1 May 2014
        Published in tosem Volume 25, Issue 1

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader