Skip to main content
SpringerLink
Log in

An empirical study on constraint optimization techniques for test generation

约束优化技术在测试用例生成中的实证研究

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Constraint solving is a frequent, but expensive operation with symbolic execution to generate tests for a program. To improve the efficiency of test generation using constraint solving, four optimization techniques are usually applied to existing constraint solvers, which are constraint independence, constraint set simplification, constraint caching, and expression rewriting. In this paper, we conducted an empirical study, using these four constraint optimization techniques in a well known test generation tool KLEE with 77 GNU Coreutils applications, to systematically investigate how these optimization techniques affect the efficiency of test generation. The experimental results show that these constraint optimization techniques as well as their combinations cannot improve the efficiency of test generation significantly for ALL-SIZED programs. Moreover, we studied the constraint optimization techniques with respect to two static metrics, lines of code (LOC) and cyclomatic complexity (CC), of programs. The experimental results show that the “constraint set simplification” technique can improve the efficiency of test generation significantly for the programs with high LOC and CC values. The “constraint caching” optimization technique can improve the efficiency of test generation significantly for the programs with low LOC and CC values. Finally, we propose four hybrid optimization strategies and practical guidelines based on different static metrics.

创新点

我们使用KLEE执行了77个GNU Coreutils程序,用于系统的研究4种流行的约束优化技术如何影响测试用例生成的效率。结果表明,对于所有程序,这些约束优化技术及它们的组合不能大幅提高测试用例生成的效率。 此外,我们研究了程序的两个静态指标(代码行和圈复杂度)跟约束优化技术的关系。结果表明,对于大规模及高圈复杂度程序,使用“约束集简化”技术,对于小规模及低圈复杂度程序,使用“约束缓存”技术,都可以显著提高测试生成效率。最后,基于不同的静态指标,我们提出了四种混合优化策略和指导方针。

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Beizer B. Software Testing Techniques. 2nd ed. New York: International Thomson Computer Press, 1990

    MATH  Google Scholar 

  2. Fang C R, Chen Z Y, Xu B W. Comparing logic coverage criteria on test case prioritization. Sci China Inf Sci, 2012, 55: 2826–2840

    Article  MathSciNet  Google Scholar 

  3. Yang R, Chen Z Y, Zhang Z Y, et al. Efsm-based test case generation: sequence, data, and oracle. Int J Softw Eng Knowl Eng, 2015, 25: 633–667

    Article  MathSciNet  Google Scholar 

  4. Orso A, Rothermel G. Software testing: a research travelogue (2000–2014). In: Proceedings of the IEEE International Conference on Future of Software Engineering (ICSE). New York: ACM, 2014. 117–132

    Google Scholar 

  5. King J C. Symbolic execution and program testing. Commun ACM, 1976, 19: 385–394

    Article  MathSciNet  MATH  Google Scholar 

  6. Chen T, Zhang X-S, Guo S-Z, et al. State of the art: dynamic symbolic execution for automated test generation. Future Gener Comput Syst, 2013, 29: 1758–1773

    Article  Google Scholar 

  7. Anand S, Burke E K, Chen T Y, et al. An orchestrated survey of methodologies for automated software test case generation. J Syst Softw, 2013, 86: 1978–2001

    Article  Google Scholar 

  8. Cadar C, Dunbar D, Engler D R. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley: USENIX Association, 2008. 209–224

    Google Scholar 

  9. Cadar C, Ganesh V, Pawlowski P M, et al. Exe: automatically generating inputs of death. ACM Trans Inf Syst Secur, 2008, 12: 10

    Article  Google Scholar 

  10. Godefroid P, Klarlund N, Sen K. Dart: directed automated random testing. In: Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. New York: ACM, 2005. 40: 213–223

    Google Scholar 

  11. Sen K, Marinov D, Agha G. CUTE: a concolic unit testing engine for C. In: Proceedings of the 10th European Software Engineering Conference Held Jointly With 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering. New York: ACM, 2005. 263–272

    Google Scholar 

  12. Barrett C, de Moura L, Stump A. Design and results of the first satisfiability modulo theories competition (smt-comp 2005). J Autom Reasoning, 2005, 35: 373–390

    Article  MATH  Google Scholar 

  13. Schittkowski K. NLPQL: a fortran subroutine solving constrained nonlinear programming problems. Ann Oper Res, 1986, 5: 485–500

    Article  MathSciNet  Google Scholar 

  14. Cadar C, Engler D. Execution generated test cases: how to make systems code crash itself. In: Model Checking Software. Berlin: Springer, 2005. 2–23

    Chapter  Google Scholar 

  15. Godefroid P, Levin M Y, Molnar D A, et al. Automated whitebox fuzz testing. In: Proceedings of the Network and Distributed System Security Symposium, San Diego, 2008. 8: 151–166

    Google Scholar 

  16. Brumley D, Newsome J, Song D, et al. Towards automatic generation of vulnerability-based signatures. In: Proceedings of IEEE Symposium on Security and Privacy, Berkeley/Oakland, 2006. 15–16

    Google Scholar 

  17. Brumley D, Newsome J, Song D, et al. Theory and techniques for automatic generation of vulnerability-based signatures. IEEE Trans Depend Secure Comput, 2008, 5: 224–241

    Article  Google Scholar 

  18. Brumley D, Wang H, Jha S, et al. Creating vulnerability signatures using weakest preconditions. In: Proceedings of the 20th IEEE Computer Security Foundations Symposium, Venice, 2007. 311–325

    Chapter  Google Scholar 

  19. Liang Z K, Sekar R. Fast and automated generation of attack signatures: a basis for building self-protecting servers. In: Proceedings of the 12th ACM Conference on Computer and Communications Security. New York: ACM, 2005. 213–222

    Google Scholar 

  20. Newsome J, Brumley D, Song D. Vulnerability-specific execution filtering for exploit prevention on commodity software. In: Proceedings of the Network and Distributed System Security Symposium, San Diego, 2006. 58–71

    Google Scholar 

  21. Brumley D, Hartwig C, Kang M G, et al. Bitscope: Automatically Dissecting Malicious Binaries. School of Computer Science, Carnegie Mellon University, Technology Report CMU-CS-07-133. 2007

    Google Scholar 

  22. Brumley D, Hartwig C, Liang Z K, et al. Automatically identifying trigger-based behavior in malware. In: Botnet Detection. Berlin: Springer, 2008. 65–88

    Chapter  Google Scholar 

  23. Moser A, Kruegel C, Kirda E. Exploring multiple execution paths for malware analysis. In: Proceedings of IEEE Symposium on Security and Privacy, Berkeley, 2007. 231–245

    Google Scholar 

  24. Song D, Brumley D, Yin H, et al. Bitblaze: a new approach to computer security via binary analysis. In: Information Systems Security. Berlin: Springer, 2008. 1–25

    Chapter  Google Scholar 

  25. Chandra A K, Iyengar V S. Constraint solving for test case generation: a technique for high-level design verification. In: Proceedings of IEEE International Conference on Computer Design: VLSI in Computers and Processors, Cambridge, 1992. 245–248

    Google Scholar 

  26. De Milli R A, Offutt A J. Constraint-based automatic test data generation. IEEE Trans Softw Eng, 1991, 17: 900–910

    Article  Google Scholar 

  27. Gotlieb A, Botella B, Rueher M. Automatic test data generation using constraint solving techniques. ACM SIGSOFT Softw Eng Notes, 1998, 23: 53–62

    Article  Google Scholar 

  28. Tovey C A. A simplified np-complete satisfiability problem. Discrete Appl Math, 1984, 8: 85–89

    Article  MathSciNet  MATH  Google Scholar 

  29. Ganesh V, Dill D L. A decision procedure for bit-vectors and arrays. In: Computer Aided Verification. Berlin: Springer, 2007. 519–531

    Chapter  Google Scholar 

  30. de Moura L, Bjørner N. Z3: an efficient smt solver. In: Tools and Algorithms for the Construction and Analysis of Systems. Berlin: Springer, 2008. 337–340

    Chapter  Google Scholar 

  31. Barrett C, Tinelli C. Cvc3. In: Computer Aided Verification. Berlin: Springer, 2007. 298–302

    Chapter  Google Scholar 

  32. Palikareva H, Cadar C. Multi-solver support in symbolic execution. In: Computer Aided Verification. Berlin: Springer, 2013. 53–68

    Chapter  Google Scholar 

  33. Jones C. Software metrics: good, bad and missing. Computer, 1994, 27: 98–100

    Article  Google Scholar 

  34. Shepperd M. A critique of cyclomatic complexity as a software metric. Softw Eng J, 1988, 3: 30–36

    Article  Google Scholar 

  35. Ferguson R, Korel B. The chaining approach for software test data generation. ACM Trans Softw Eng Meth, 1996, 5: 63–86

    Article  Google Scholar 

  36. Offutt A J, Hayes J H. A semantic model of program faults. ACM SIGSOFT Soft Eng Notes, 1996, 21: 195–200

    Article  Google Scholar 

  37. Brummayer R, Biere A. Boolector: an efficient smt solver for bit-vectors and arrays. In: Tools and Algorithms for the Construction and Analysis of Systems. Berlin: Springer, 2009. 174–177

    Chapter  Google Scholar 

  38. Erete I, Orso A. Optimizing constraint solving to better support symbolic execution. In: Proceedings of IEEE 4th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Berlin, 2011. 310–315

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210093, China

    Zhiyi Zhang, Zhenyu Chen & Baowen Xu

  2. Department of Computer Science, University of Texas at Dallas, Richardson, 75080-3021, USA

    Ruizhi Gao & Eric Wong

Authors
  1. Zhiyi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruizhi Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Baowen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenyu Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Chen, Z., Gao, R. et al. An empirical study on constraint optimization techniques for test generation. Sci. China Inf. Sci. 60, 012105 (2017). https://doi.org/10.1007/s11432-015-0450-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-015-0450-5

Keywords

Search

Navigation