Yunho Kim
ABSTRACT
Concolic testing is popular in unit testing because it can detect bugs quickly in a relatively small search space. But, in system-level testing, it suffers from the symbolic path explosion and often misses bugs. To resolve this problem, we have developed a focused compositional concolic testing technique, FOCAL, for effective bug detection. Focusing on a target unit failure v (a crash or an assert violation) detected by concolic unit testing, FOCAL generates a system-level test input that validates v. This test input is obtained by building and solving symbolic path formulas that represent system-level executions raising v. FOCAL builds such formulas by combining function summaries one by one backward from a function that raised v to main. If a function summary φa of function a conflicts with the summaries of the other functions, FOCAL refines φa to φa′ by applying a refining constraint learned from the conflict. FOCAL showed high system-level bug detection ability by detecting 71 out of the 100 real-world target bugs in the SIR benchmark, while other relevant cutting edge techniques (i.e., AFL-fast, KATCH, Mix-CCBSE) detected at most 40 bugs. Also, FOCAL detected 13 new crash bugs in popular file parsing programs.
- Leonardo Alt, Sepideh Asadi, Hana Chockler, Karine Even Mendoza, Grigory Fedyukovich, Antti E. J. Hyvärinen, and Natasha Sharygina. 2017. HiFrog: SMTbased Function Summarization for Software Verification. In Tools and Algorithms for the Construction and Analysis of Systems, Axel Legay and Tiziana Margaria (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 207–213. Google Scholar
Digital Library
- Sepideh Asadi, Martin Blicha, Grigory Fedyukovich, Antti Hyv\"arinen, Karine Even-Mendoza, Natasha Sharygina, and Hana Chockler. 2018.Google Scholar
- Function Summarization Modulo Theories. In LPAR-22. 22nd International Conference on Logic for Programming, Artificial Intelligence and Reasoning (EPiC Series in Computing), Gilles Barthe, Geoff Sutcliffe, and Margus Veanes (Eds.), Vol. 57. EasyChair, 56–75.Google Scholar
- Nels E. Beckman, Aditya V. Nori, Sriram K. Rajamani, and Robert J. Simmons. 2008.Google Scholar
- Proofs from Tests. In Proceedings of the 2008 International Symposium on Software Testing and Analysis (ISSTA ’08). ACM, New York, NY, USA, 3–14.Google Scholar
- Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coveragebased Greybox Fuzzing As Markov Chain. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16). ACM, New York, NY, USA, 1032–1043. Google ScholarDigital Library
- Tim Bray. 2017. The JavaScript Object Notation ( JSON) Data Interchange Format. RFC 8259.Google Scholar
- Jacob Burnim and Koushik Sen. 2008.Google Scholar
- Heuristics for Scalable Dynamic Test Generation. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE ’08). IEEE Computer Society, Washington, DC, USA, 443–446. Google ScholarDigital Library
- Arindam Chakrabarti and Patrice Godefroid. 2006. Software Partitioning for Effective Automated Unit Testing. In Proceedings of the 6th International Conference on Embedded Software (EMSOFT ’06). ACM, New York, NY, USA, 262–271. Google ScholarDigital Library
- William Craig. 1957. Three Uses of the Herbrand-Gentzen Theorem in Relating Model Theory and Proof Theory. The Journal of Symbolic Logic 22, 3 (1957), 269–285. http://www.jstor.org/stable/2963594Google ScholarCross Ref
- Leonardo De Moura and Nikolaj Bjørner. 2008.Google Scholar
- Peter Dinges and Gul Agha. 2014. Targeted Test Input Generation Using Symbolicconcrete Backward Execution. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14). ACM, New York, NY, USA, 31–36. Google ScholarDigital Library
- Hyunsook Do, Sebastian Elbaum, and Gregg Rothermel. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact. Empirical Software Engineering 10, 4 (Oct. 2005), 405–435. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2013. 1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite. Empirical Software Engineering 20, 3 (2013), 611–639. Google ScholarDigital Library
- Patrice Godefroid. 2007. Compositional Dynamic Test Generation. In Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’07). ACM, New York, NY, USA, 47–54. Google ScholarDigital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’05). ACM, New York, NY, USA, 213–223. Google ScholarDigital Library
- Patrice Godefroid, Aditya V. Nori, Sriram K. Rajamani, and Sai Deep Tetali. 2010.Google Scholar
- Compositional May-must Program Analysis: Unleashing the Power of Alternation. In Proceedings of the 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’10). ACM, New York, NY, USA, 43–56. Google ScholarDigital Library
- Florian Gross, Gordon Fraser, and Andreas Zeller. 2012. Search-Based System Testing: High Coverage, No False Alarms. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA ’12). ACM, New York, NY, USA, 67–77. Google ScholarDigital Library
- Shin Hong, Taehoon Kwak, Byeongcheol Lee, Yiru Jeon, Bongseok Ko, Yunho Kim, and Moonzoo Kim. 2017. MUSEUM: Debugging real-world multilingual programs using mutation analysis. Information and Software Technology 82 (2017), 80 – 95.Google ScholarCross Ref
- Joxan Jaffar, Vijayaraghavan Murali, and Jorge A. Navas. 2013. Boosting Concolic Testing via Interpolation. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). ACM, New York, NY, USA, 48–58. Google ScholarDigital Library
- Wei Jin and Alessandro Orso. 2012. BugRedux: Reproducing Field Failures for Inhouse Debugging. In Proceedings of the 34th International Conference on Software Engineering (ICSE ’12). IEEE Press, Piscataway, NJ, USA, 474–484. http://dl.acm. org/citation.cfm?id=2337223.2337279 Google ScholarDigital Library
- Moonzoo Kim, Yunho Kim, and Yunja Choi. 2012. Concolic testing of the multisector read operation for flash storage platform software. Formal Aspects of Computing 24, 3 (01 May 2012), 355–374. 011- 0200- 9Google Scholar
- M. Kim, Y. Kim, and H. Kim. 2011.Google Scholar
- Comparative Study on Software Model Checkers as Unit Testing Tools: An Industrial Case Study. IEEE Transactions on Software Engineering (TSE) 37, 2 (March 2011), 146–160. Google ScholarDigital Library
- Yunho Kim, Yunja Choi, and Moonzoo Kim. 2018. Precise Concolic Unit Testing of C Programs Using Extended Units and Symbolic Alarm Filtering. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). ACM, New York, NY, USA, 315–326. Google ScholarDigital Library
- Yunho Kim, Shin Hong, Bongseok Ko, Duy Loc Phan, and Moonzoo Kim. 2018. Invasive Software Testing: Mutating Target Programs to Diversify Test Exploration for High Test Coverage. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation.Google ScholarCross Ref
- Yunho Kim and Moonzoo Kim. {n.d.}. CROWN: Concolic testing for Real-wOrld softWare aNalysis. http://github.com/swtvkaist/CROWN Accessed: 2019-06-29.Google Scholar
- Yunho Kim, Youil Kim, Taeksu Kim, Gunwoo Lee, Yoonkyu Jang, and Moonzoo Kim. 2013. Automated Unit Testing of Large Industrial Embedded Software Using Concolic Testing. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE’13). IEEE Press, Piscataway, NJ, USA, 519–528. Google ScholarDigital Library
- Yunho Kim, Dongju Lee, Junki Baek, and Moonzoo Kim. 2019. Concolic Testing for High Test Coverage and Reduced Human Effort in Automotive Industry. In International Conference on Software Engineering (ICSE) Software Engineering In Practice (SEIP) track. Google ScholarDigital Library
- Chris Lattner and Vikram Adve. 2004.Google Scholar
- Kin-Keung Ma, Khoo Yit Phang, Jeffrey S. Foster, and Michael Hicks. 2011.Google ScholarDigital Library
- Directed Symbolic Execution. In Proceedings of the 18th International Conference on Static Analysis (SAS’11). Springer-Verlag, Berlin, Heidelberg, 95–111. http://dl.acm.org/citation.cfm?id=2041552.2041563 Google ScholarDigital Library
- Paul Dan Marinescu and Cristian Cadar. 2013. KATCH: High-coverage Testing of Software Patches. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). ACM, New York, NY, USA, 235–245. Google ScholarDigital Library
- Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014.Google Scholar
- Ask the Mutants: Mutating Faulty Programs for Fault Localization. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation (ICST ’14). IEEE Computer Society, Washington, DC, USA, 153–162. Google ScholarDigital Library
- Van-Thuan Pham, Wei Boon Ng, Konstantin Rubinov, and Abhik Roychoudhury. 2015.Google Scholar
- Hercules: Reproducing Crashes in Real-world Application Binaries. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE ’15). IEEE Press, Piscataway, NJ, USA, 891–901. http://dl.acm.org/citation. cfm?id=2818754.2818862 Google ScholarDigital Library
- Rui Qiu, Guowei Yang, Corina S. Păsăreanu, and Sarfraz Khurshid. 2015. Compositional Symbolic Execution with Memoized Replay. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE ’15). IEEE Press, Piscataway, NJ, USA, 632–642. http://dl.acm.org/citation.cfm?id=2818754.2818832 Google ScholarDigital Library
- Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13). ACM, New York, NY, USA, 263–272. Google ScholarCross Ref
- Ondrej Sery, Grigory Fedyukovich, and Natasha Sharygina. 2011. Interpolation-Based Function Summaries in Bounded Model Checking. In Hardware and Software: Verification and Testing, Kerstin Eder, João Lourenço, and Onn Shehory (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 160–175. Google ScholarDigital Library
- Sina Shamshiri, Rene Just, Jose Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (ASE ’15). IEEE Computer Society, Washington, DC, USA, 201–211.Google ScholarDigital Library
- Nishant Sinha, Nimit Singhania, Satish Chandra, and Manu Sridharan. 2012.Google Scholar
- Alternate and Learn: Finding Witnesses without Looking All over. In Computer Aided Verification, P. Madhusudan and Sanjit A. Seshia (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 599–615. Google ScholarDigital Library
- Nikolai Tillmann and Jonathan De Halleux. 2008. Pex: White Box Test Generation for .NET. In Proceedings of the 2Nd International Conference on Tests and Proofs (TAP’08). Springer-Verlag, Berlin, Heidelberg, 134–153. Google ScholarDigital Library
Index Terms
- Target-driven compositional concolic testing with function summary refinement for effective bug detection
Recommendations
Concolic testing with adaptively changing search heuristics
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringWe present Chameleon, a new approach for adaptively changing search heuristics during concolic testing. Search heuristics play a central role in concolic testing as they mitigate the path-explosion problem by focusing on particular program paths that ...
Automated Test Generation Using Concolic Testing
ISEC '15: Proceedings of the 8th India Software Engineering ConferenceIn this talk, I will talk about the recent advances and challenges in concolic testing and symbolic execution. Concolic testing, also known as directed automated random testing (DART) or dynamic symbolic execution, is an efficient way to automatically ...
Differential testing: a new approach to change detection
ESEC-FSE companion '07: The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papersRegression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression ...
Comments