Abstract
Although many test case prioritization techniques exist, their performance is far from perfect. Hence, we propose a new fault-based test case prioritization technique to promote fault-revealing test cases in model-based testing (MBT) procedures. We seek to improve the fault detection rate—a measure of how fast a test suite is able to detect faults during testing—in scenarios such as regression testing. We propose an extended digraph model as the basis of this new technique. The model is realized using a novel reinforcement-learning (RL)- and hidden-Markov-model (HMM)-based technique which is able to prioritize test cases for regression testing objectives. We present a method to initialize and train an HMM based upon RL concepts applied to an application's digraph model. The model prioritizes test cases based upon forward probabilities, a new test case prioritization approach. In addition, we also propose an alternative approach to prioritizing test cases according to the amount of change they cause in applications. To evaluate the effectiveness of the proposed techniques, we perform experiments on graphical user interface (GUI)-based applications and compare the results with state-of-the-art test case prioritization approaches. The experimental results show that the proposed technique is able to detect faults early within test runs.
- K. Aas and L. Eikvil. 1996. Text page recognition using grey-level features and hidden Markov models. Patt. Recogn. 29, 6, 977--985.Google ScholarCross Ref
- A. Arcuri and L. Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (ICSE'11). IEEE, 1--10. Google ScholarDigital Library
- I. Arel, C. Liu, T. Urbanik, and A. G. Kohls. 2010. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intell. Transport Syst. 4, 2, 128--135.Google ScholarCross Ref
- D. Ariu, R. Tronci, and G. Giacinto. 2011. HMMPayl: An intrusion detection system based on hidden Markov models. Comput. Secur. 30, 4, 221--241. Google ScholarDigital Library
- S. Arlt, S. Pahl, C. Berolini, and M. Schaf. 2012. Trends in model-based GUI testing. Adv. Comput. 86, 183--222.Google ScholarCross Ref
- A. Avritzer and E. R. Weyuker. 1995. The automatic generation of load test suites and the assessment of the resulting software. IEEE Trans. Softw. Engin. 21, 9, 705--716. Google ScholarDigital Library
- C. G. Bai, Q. P. Hu, M. Xie, and S. H. Ng. 2005. Software failure prediction based on a Markov Bayesian network model. J. Syst. Softw. 74, 3, 275--282. Google ScholarDigital Library
- L. Baum, T. Petrie, G. Soules, and N. Weiss. 1970. A maximization technique occuring in the statistical analysis of probabilistic functions of Markov chains. The Ann. Math. Statist. 41, 1, 164--171.Google ScholarCross Ref
- G. Becce, L. Mariani, O. Riganelli, and M. Santoro. 2012. Extracting widget descriptions from GUIs. In Proceedings of the 15th International Conference on Fundamental Approaches to Software Engineering (FASE'12). 347--361. Google ScholarDigital Library
- P. Blunsom. 2004. Hidden Markov models. http://digital.cs.usu.edu/∼cyan/CS7960/hmm-tutorial.pdf.Google Scholar
- K.-Y. Cai, Y.-C. Li, and W.-Y. Ning. 2005. Optimal software testing in the setting of controlled Markov chains. Euro. J. Oper. Res. 162, 2, 552--579.Google ScholarCross Ref
- H. S. Chang. 2006. Reinforcement learning with supervision by combining multiple learnings and expert advices. In Proceedings of the American Control Conference (ACC'06). 159--161.Google Scholar
- W. Ching. 2006. Markov Chains: Model, Algorithms and Applications. Springer Science. Google ScholarDigital Library
- J. Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associate Publishers.Google Scholar
- H. Cua and N. Dethlefs. 2011. Hierarchical reinforcement learning and hidden Markov models for task-oriented natural language generation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short Papers (ACL'11). 654--659. Google ScholarDigital Library
- R. Davoodi and B. J. Andrews. 1998. Computer simulation of FES standing up in paraplegia: A self-adaptive fuzzy controller with reinforcement learning. IEEE Trans. Rehabil. Engin. 6, 2, 151--161.Google ScholarCross Ref
- P. Dayan and C. J. Watkin. 1992. Technical note Q-learning. Mach. Learn. 8, 279--292. Google ScholarDigital Library
- R. Dev, A. Jaaskelainen, and M. Katara. 2012. Model-based GUI testing: Case smartphone camera and messaging development. Adv. Comput. 85, 65--122.Google ScholarCross Ref
- H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel. 2010. The effect of time constraints in test case prioritization: A series of controlled experiments. IEEE Trans. Softw. Engin. 36, 5, 592--617. Google ScholarDigital Library
- I. K. El-Far and J. A. Whittaker. 2002. Model-based software testing. In Encyclopedia of Software Engineering. Wiley, 1--22.Google Scholar
- S. Elbaum, G. Rothermel, A. G. Malishevsky, and S. Member. 2002. Test case prioritization: A family of empirical studies test case prioritization. IEEE Trans. Softw. Engin. 27, 10, 929--948.Google Scholar
- D. P. Ellis. 2010. The Essential Guide to Effect Sizes; Statistical Power, Meta-Analysis and the Interpretation of Research Results. Cambridge University Press.Google Scholar
- A. T. Endo and A. Simao. 2011. Model-based testing of service-oriented applications via state models. In Proceedings of the IEEE International Conference on Services Computing (SCC'11). 432--439. Google ScholarDigital Library
- U. Farooq. 2011. Model based test suite minimization using metaheuristics. http://ro.ecu.edu.au/cgi/viewcontent.cgi?articlen=1409&context==theses.Google Scholar
- A. Feliachi and H. L. Guen. 2010. Generating transition probabilities for automatic model-based test generation. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation (ICST'10). 99--102. Google ScholarDigital Library
- P. G. Frankl, G. Rothermel, K. Sayre, and F. I. Vokolos. 2003. An empirical comparison of two safe regression test selection techniques. In Proceedings of the International Symposium on Empirical Software Engineering (ISESE'03). IEEE Computer Society, 195--204. Google ScholarDigital Library
- F. Galon. 1889. Natural Inheritance. MacMillan.Google Scholar
- K. Hamamoto, K. Morooka, and H. Nagahashi. 2004. Motion recognition by combining HMM and reinforcement learning. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (ICSMC'04). 259--264.Google Scholar
- J. Hao and E. Mendes. 2006. Usage-based statistical testing of Web applications. In Proceedings of the 6th International Conference on Web Engineering (ICWE'06). ACM Press, New York, 17--24. Google ScholarDigital Library
- H. Hemmati. 2011. Similarity-based test case selection: Toward scalable and practical model-based testing. https://www.simula.no/publications/similarity-based-test-case-selection-toward-scalable-and-practical-model-based-testing.Google Scholar
- H. Hemmati, A. Arcuri, and L. Briand. 2013. Achieving scalable model-based testing through test case diversity. ACM Trans. Softw. Engin. Methodol. 22, 1, 1--42. Google ScholarDigital Library
- M. A. T. Ho, Y. Yamada, and Y. Umetani. 2002. An HMM-based temporal difference learning with model-updating capability for visual tracking of human communicational behaviors. In Proceedings of the 5th IEEE International Conference on Automatic Face Gesture Recognition (FGR'02). 170--175. Google ScholarDigital Library
- H. Hsu and P. A. Lachenbruch. 2008. Paired t-test. In Encyclopedia of Clinical Trials. Wiley, 1--3.Google Scholar
- C.-Y. Huang, J.-R. Chang, and Y.-H. Chang. 2010. Design and analysis of GUI test-case prioritization using weight-based methods. J. Syst. Softw. 83, 4, 646--659. Google ScholarDigital Library
- R. Ihaka and R. Gentleman. 1996. R: A language for data analysis and graphics. J. Comput. Graph. Statist. 5, 3, 299--314.Google ScholarCross Ref
- T. P. Jaakkola, S. P. Singh, and M. I. Jordan. 1995. Reinforcement learning algorithm for partially observable Markov decision problems. http://papers.nips.cc/paper/951-reinforcement-learning-algorithm-for-partially-observable-markov-decision-problems.pdf.Google Scholar
- D. Jeffrey and N. Gupta. 2007. Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Trans. Softw. Engin. 33, 2, 108-123. Google ScholarDigital Library
- J. Kabudian, M. R. Meybodi, and M. M. Homayounpour. 2004. Applying continuous action reinforcement learning automata (CARLA) to global training of hidden Markov models. In Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'04). 638-642. Google ScholarDigital Library
- K. Lee, H. Hon, M. Hwang, and X. Huang. 1990. Speech recognition using hidden Markov model: A CMU perspective. Speech Comm. 9, 5--6, 497--508. Google ScholarDigital Library
- B. Legeard and M. Utting. 2006. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, San Fransisco. Google ScholarDigital Library
- L. J. Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn. 8, 3--4, 293--321. Google ScholarDigital Library
- F. Lindlar, A. Windisch, and J. Wegener. 2010. Integrating model-based testing with evolutionary functional testing. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW'10). 163--172. Google ScholarDigital Library
- L. Mariani, M. Pezz, O. Riganelli, and M. Santoro. 2011. AutoBlackTest: A tool for automatic black-box testing. In Proceedings of the 33rd International Conference on Software Engineering (ICSE'11). 1013--1015. Google ScholarDigital Library
- L. Mariani, M. Pezz, O. Riganelli, and M. Santoro. 2012. AutoBlackTest: Automatic black-box testing of interactive applications. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST'12). 81--90. Google ScholarDigital Library
- A. K. McCallum. 1995. Reinforcement learning with selective percemption and hidden state. Doctoral dissertation, University of Rochester, NY. http://web.media.mit.edu/∼tristan/Classes/MAS.945/Papers/Contextual/McCallum_Thesis.pdf. Google ScholarDigital Library
- M. McPartland and M. Gallagher. 2011. Reinforcement learning in first person shooter games. IEEE Trans. Comput. Intell. AI Games 3, 1, 43--56.Google ScholarCross Ref
- H. Mei, D. Hao, L. Zhang, J. Zhou, and G. Rothermel. 2012. A static approach to prioritizing J-unit test cases. IEEE Trans. Softw. Engin. 38, 6, 1258--1275. Google ScholarDigital Library
- A. M. Memon. 2007. An event-flow model of GUI-based applications for testing. Softw. Test. Verif. Reliab. 17, 3, 137--157. Google ScholarDigital Library
- A. M. Memon and D. R. Hackner. 2008. Test case generator for GUITAR. In Proceedings of the Companion to the International Conference of Software Engineering (ICSE'08). 959--960. Google ScholarDigital Library
- S. Mirarab and L. Tahvildari. 2007. A prioritization approach for software. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE'07). 276--290. Google ScholarDigital Library
- S. Mirarab and L. Tahvildari. 2008. An empirical study on Bayesian network-based approach for test case prioritization. In Proceedings of the 1st International Conference on Software Testing, Verification, and Validation (ICST'08). 278--287. Google ScholarDigital Library
- D. Nardo, N. Alshahwan, and L. Briand. 2013. Coverage-based test case prioritisation: An industrial case study. In Proceeding of the 6th International Conference on Software Testing, Verification, and Validation (ICST'13). 302--311. Google ScholarDigital Library
- S. Parsa and A. Khalilian. 2010. On the optimization approach towards test suite minimization approach. Int. J. Softw. Engin. Appl. 4, 1, 15--28.Google Scholar
- L. R. Rabiner and B. H. Juang. 1986. An introduction to hidden Markov models. IEEE ASSP Mag. 3, 1, 4--16.Google ScholarCross Ref
- H. Reza, S. Endapally, and E. Grant. 2007. A model-based approach for testing GUI using hierarchical predicate transition nets. In Proceedings of the 4th International Conference on Information Technology (ITNG'07). 366--370. Google ScholarDigital Library
- J. A. Rice. 1994. Mathematical Statistics and Data Analysis. Duxbury Press.Google Scholar
- M. Robinson and P. A. Vorobiev. 1999. Swing: A Fast-Paced Guide with Production-Quality Code Examples. Manning. Google ScholarDigital Library
- G. Rothermel and D. Hall. 1997. A safe, efficient regression test selection technique. ACM Trans. Softw. Engin. Methodol. 6, 2, 173--210. Google ScholarDigital Library
- G. Rothermel and M. J. Harrold. 1998. Empirical studies of a safe regression test selection technique. IEEE Trans. Softw. Engin. 24, 6, 108--123. Google ScholarDigital Library
- G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the International Conference on Software Maintenance (ICSM'98). 34--43. Google ScholarDigital Library
- G. Rothermel, R. Untch, and M. J. Harrold. 2001. Prioritizing test cases for regression testing. IEEE Trans. Softw. Engin. 27, 10, 929--948. Google ScholarDigital Library
- G. S. Semmel and D. G. Linton. 1998. Determining optimal testing times for Markov chain usage models. In Proceedings of the IEEE SECON Conference (Southeastcon'98). 1--4.Google Scholar
- R. S. Sutton and A. G. Barto. 1998. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, Massachusetts. Google ScholarDigital Library
- C. Szepesvári. 2010. Algorithms for Reinforcement Learning. Morgan and Claypool. Google ScholarDigital Library
- S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein. 2012. Static test case prioritization using topic models. Empir. Softw. Engin. 19, 1, 182--212. Google ScholarDigital Library
- O. Tramasco and S. Bauer. 2013. Package ‘RHmm’. http://www2.uaem.mx/r-mirror/web/packages/RHmm/RHmm.pdf.Google Scholar
- M. A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. J. Artif. Intell. Res. 12, 387--416. Google ScholarCross Ref
- C. J. Watkin. 1989. Learning from delayed rewards. 1--14. https://www.cs.rhul.ac.uk/home/chrisw/new_thesis.pdf.Google Scholar
- A. L. White and J. A. Sjorgen. 1988. Markov chains for testing redundant software. In Proceedings of the Reliability and Maintainability Symposium (RAMS'88). 426--433.Google Scholar
- J. A. Whittaker and M. G. Thomason. 1994. A Markov chain model for statistical software testing. IEEE Trans. Softw. Engin. 20, 10, 812--824. Google ScholarDigital Library
- M. Wiering and M. Van Otterlo. 2012. Reinforcement Learning (State of the Art). Springer. Google ScholarDigital Library
- J. D. Williams and S. Young. 2007. Partially observable Markov decision processes for spoken dialog systems. Comput. Speech Lang. 21, 2, 393--422. Google ScholarDigital Library
- W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. 1997. A study of effective regression testing in practice. In Proceedings of the 8th IEEE International Symposium on Software Reliability Engineering (ISSRE'97). 264--274. Google ScholarDigital Library
- M. M. Yin and J. T. L. Wang. 2001. Effective hidden Markov models for detecting splicing junction sites in DNA sequences. Inf. Sci. 139, 1--2, 139--163. Google ScholarDigital Library
- S. Yoo and M. Harman. 2007. Regression testing minimisation, selection and prioritisation: A survey. Softw. Test. Verif. Reliab. 7, 1--60. Google ScholarDigital Library
- Y. T. Yu and M. F. Lau. 2012. Fault-based test suite prioritization for specification-based testing. Inf. Softw. Technol. 54, 2, 179--202. Google ScholarDigital Library
- L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei. 2009a. Time-aware test-case prioritization using integer linear programming. In Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA'09). ACM Press, New York, 401--419. Google ScholarDigital Library
- Z. Zhang, W. K. Chan, and T. H. Tse. 2009b. Adaptive random test case prioritization. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE'09). Google ScholarDigital Library
- F. Zhen And P. Chenglian. 2004. A system test methodology based on the Markov chain usage model. In Proceedings of the 8th International Conference on Computer Supported Cooperative Work in Design (CACWD'04). 160--165.Google Scholar
Index Terms
- Test Case Prioritization Using Extended Digraphs
Recommendations
Adaptive random prioritization for interaction test suites
SAC '14: Proceedings of the 29th Annual ACM Symposium on Applied ComputingCombinatorial interaction testing (CIT), a black-box testing method, has been well studied in recent years. It aims at constructing an effective interaction test suites, so as to identify the faults that are caused by interactions among parameters. ...
Multi-objective test case prioritization for GUI applications
SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied ComputingTest case prioritization techniques are proposed to schedule execution of test cases in order to improve testing effectiveness. Various coverage criteria are used as surrogates for test case prioritization. They are expected to improve testing ...
Static test case prioritization using topic models
Software development teams use test suites to test changes to their source code. In many situations, the test suites are so large that executing every test for every source code change is infeasible, due to time and resource constraints. Development ...
Comments