Skip to main content

Advertisement

SpringerLink
Log in

DeltaFuzz: Historical Version Information Guided Fuzz Testing

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

With the widespread use of agile software development methods, such as agile and scrum, software is iteratively updated more frequently. To ensure the quality of the software, regression testing is conducted before new versions are released. Moreover, to improve the efficiency of regression testing, testing efforts should be concentrated on the modified and impacted parts of a program. However, the costs of manually constructing new test cases for the modified and impacted parts are relatively expensive. Fuzz testing is an effective method for generating test data automatically, but it is usually devoted to achieving higher code coverage, which makes fuzz testing unsuitable for direct regression testing scenarios. For this reason, we propose a fuzz testing method based on the guidance of historical version information. First, the differences between the program being tested and the last version are analyzed, and the results of the analysis are used to locate change points. Second, change impact analysis is performed to find the corresponding impacted basic blocks. Finally, the fitness values of test cases are calculated according to the execution traces, and new test cases are generated iteratively by the genetic algorithm. Based on the proposed method, we implement a prototype tool DeltaFuzz and conduct experiments on six open-source projects. Compared with the fuzzing tool AFLGo, AFLFast and AFL, DeltaFuzz can reach the target faster, and the time taken by DeltaFuzz was reduced by 20.59%, 30.05% and 32.61%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Masso J, Pino F J, Pardo C et al. Risk management in the software life cycle: A systematic literature review. Computer Standards & Interfaces, 2020, 71: Article No. 103431. https://doi.org/10.1016/j.csi.2020.103431.

  2. Gu T, Ma X, Xu C et al. Synthesizing object transformation for dynamic software updating. In Proc. the 39th IEEE/ACM International Conference on Software Engineering Companion, May 2017, pp.336-338. https://doi.org/10.1109/ICSE-C.2017.96.

  3. Khatibsyarbini M, Isa M A, Jawawi D N A et al. Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology, 2018, 93: 74-93. https://doi.org/10.1016/j.infsof.2017.08.014.

    Article  Google Scholar 

  4. Han J C, Zhou Z Q. Metamorphic fuzz testing of autonomous vehicles. In Proc. the 42nd IEEE/ACM International Conference on Software Engineering, June 27–July 19, 2020, pp.380-385. https://doi.org/10.1145/3387940.3392252.

  5. Dong Z, Böhme M, Cojocaru L et al. Time-travel testing of Android apps. In Proc. the 42nd IEEE/ACM International Conference on Software Engineering, June 27–July 19, 2020, pp.481-492. https://doi.org/10.1145/3377811.3380402.

  6. Qian J, Zhou D. Prioritizing test cases for memory leaks in Android applications. Journal of Computer Science and Technology, 2016, 31(5): 869-882. https://doi.org/10.1007/s11390-016-1670-2.

    Article  Google Scholar 

  7. Chen Y, Su T, Su Z. Deep differential testing of JVM implementations. In Proc. the 41st IEEE/ACM International Conference on Software Engineering, May 2019, pp.1257-1268. https://doi.org/10.1109/ICSE.2019.00127.

  8. Wüstholz V, Christakis M. Targeted greybox fuzzing with static lookahead analysis. In Proc. the 42nd IEEE/ACM International Conference on Software Engineering, June 27–July 19, 2020, pp.789-800. https://doi.org/10.1145/3377811.3380388.

  9. Böhme M, Manès V J M, Cha S K. Boosting fuzzer efficiency: An information theoretic perspective. In Proc. the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, November 2020, pp.678-689. https://doi.org/10.1145/3368089.3409748.

  10. Song S, Song C, Jang Y et al. CrFuzz: Fuzzing multipurpose programs through input validation. In Proc. the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, November 2020, pp.690-700. https://doi.org/10.1145/3368089.3409769.

  11. Havrikov N. Efficient fuzz testing leveraging input, code, and execution. In Proc. the 39th IEEE/ACM International Conference on Software Engineering Companion, May 2017, pp.417-420. https://doi.org/10.1109/ICSE-C.2017.26.

  12. Klees G, Ruef A, Cooper B et al. Evaluating fuzz testing. In Proc. the 2018 ACM SIGSAC Conference on Computer and Communications Security, October 2018, pp.2123-2138. https://doi.org/10.1145/3243734.3243804.

  13. Zhang M Z, Gong Y Z, Wang YW et al. Unit test data generation for C using rule-directed symbolic execution. Journal of Computer Science and Technology, 2019, 34(3): 670-689. https://doi.org/10.1007/s11390-019-1935-7.

  14. He J, Balunović M, Ambroladze N et al. Learning to fuzz from symbolic execution with application to smart contracts. In Proc. the 2019 ACM SIGSAC Conference on Computer and Communications Security, November 2019, pp.531-548. https://doi.org/10.1145/3319535.3363230.

  15. Zhang Q, Wang J, Gulzar M A et al. BigFuzz: Efficient fuzz testing for data analytics using framework abstraction. In Proc. the 35th IEEE/ACM International Conference on Automated Software Engineering, September 2020, pp.722-733. https://doi.org/10.1145/3324884.3416641.

  16. Nguyen H L, Nassar N, Kehrer T et al. MoFuzz: A fuzzer suite for testing model-driven software engineering tools. In Proc. the 35th IEEE/ACM International Conference on Automated Software Engineering, September 2020, pp.1103-1115. https://doi.org/10.1145/3324884.3416668.

  17. Olsthoorn M, Van Deursen A, Panichella A. Generating highly-structured input data by combining search-based testing and grammar-based fuzzing. In Proc. the 35th IEEE/ACM International Conference on Automated Software Engineering, September 2020, pp.1224-1228. https://doi.org/10.1145/3324884.3418930.

  18. Nguyen T D, Pham L H, Sun J et al. sfuzz: An efficient adaptive fuzzer for solidity smart contracts. In Proc. the 42nd ACM/IEEE International Conference on Software Engineering, June 27–July 19, 2020, pp.778-788. https://doi.org/10.1145/3377811.3380334.

  19. Manès V J M, Kim S, Cha S K. Ankou: Guiding greybox fuzzing towards combinatorial difference. In Proc. the 42nd ACM/IEEE International Conference on Software Engineering, June 27–July 19, 2020, pp.1024-1036. https://doi.org/10.1145/3377811.3380421.

  20. Böhme M, Pham V T, Nguyen M D et al. Directed greybox fuzzing. In Proc. the 2017 ACM SIGSAC Conference on Computer and Communications Security, October 30–November 3, 2017, pp.2329-2344. https://doi.org/10.1145/3133956.3134020.

  21. Gao X, Saha R K, Prasad M R et al. Fuzz testing based data augmentation to improve robustness of deep neural networks. In Proc. the 42nd IEEE/ACM International Conference on Software Engineering, June 27–July 19, 2020, pp.1147-1158. https://doi.org/10.1145/3377811.3380415.

  22. Wen C, Wang H, Li Y et al. MemLock: Memory usage guided fuzzing. In Proc. the 42nd ACM/IEEE International Conference on Software Engineering, June 27–July 19, 2020, pp.765-777. https://doi.org/10.1145/3377811.3380396.

  23. Babić D, Bucur S, Chen Y et al. FUDGE: Fuzz driver generation at scale. In Proc. the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2019, pp.975-985. https://doi.org/10.1145/3338906.3340456.

  24. Chen H, Xue Y, Li Y et al. Hawkeye: Towards a desired directed grey-box fuzzer. In Proc. the 2018 ACM SIGSAC Conference on Computer and Communications Security, October 2018, pp.2095-2108. https://doi.org/10.1145/3243734.3243849.

  25. Medicherla R K, Komondoor R, Roychoudhury A. Fitness guided vulnerability detection with greybox fuzzing. In Proc. the 42nd IEEE/ACM International Conference on Software Engineering, June 27–July 19, 2020, pp.513-520. https://doi.org/10.1145/3387940.3391457.

  26. Österlund S, Razavi K, Bos H et al. ParmeSan: Sanitizer-guided greybox fuzzing. In Proc. the 29th USENIX Security Symposium, August 2020, pp.2289-2306.

  27. Gao X, Mechtaev S, Roychoudhury A. Crash-avoiding program repair. In Proc. the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, July 2019, pp.8-18. DOI: https://doi.org/10.1145/3293882.3330558.

  28. Fioraldi A, Maier D, Eißfeldt H et al. AFL++: Combining incremental steps of fuzzing research. In Proc. the 14th USENIX Workshop on Offensive Technologies, August 2020.

  29. Grieco G, Ceresa M, Buiras P. QuickFuzz: An automatic random fuzzer for common file formats. ACM SIGPLAN Notices, 2016, 51(12): 13-20. https://doi.org/10.1145/2976002.2976017.

    Article  Google Scholar 

  30. Liang H, Zhang Y, Yu Y et al. Sequence coverage directed greybox fuzzing. In Proc. the 27th IEEE/ACM International Conference on Program Comprehension, May 2019, pp.249-259. https://doi.org/10.1109/ICPC.2019.00044.

  31. Zhang M, Liu J, Ma F et al. IntelliGen: Automatic driver synthesis for fuzz testing. In Proc. the 43rd IEEE/ACM International Conference on Software Engineering, May 2021, pp.318-327. https://doi.org/10.1109/ICSESEIP52600.2021.00041.

  32. You W, Liu X, Ma S et al. SLF: Fuzzing without valid seed inputs. In Proc. the 41st IEEE/ACM International Conference on Software Engineering, May 2019, pp.712-723. https://doi.org/10.1109/ICSE.2019.00080.

  33. Choi W, Sen K, Necul G et al. DetReduce: Minimizing Android GUI test suites for regression testing. In Proc. the 40th IEEE/ACM International Conference on Software Engineering, May 27–June 3, 2018, pp.445-455. https://doi.org/10.1145/3180155.3180173.

  34. Zhang L. Hybrid regression test selection. In Proc. the 40th IEEE/ACM International Conference on Software Engineering, May 27–June 3, 2018, pp.199-209. DOI: 10.1145/3180155.3180198.

  35. Nagy S, Hicks M. Full-speed fuzzing: Reducing fuzzing overhead through coverage-guided tracing. In Proc. the 2019 IEEE Symposium on Security and Privacy, May 2019, pp.787-802. https://doi.org/10.1109/SP.2019.00069.

  36. Liang J, Jiang Y, Wang M et al. DeepFuzzer: Accelerated deep greybox fuzzing. IEEE Transactions on Dependable and Secure Computing, 2019, 18(6): 2675-2688. https://doi.org/10.1109/TDSC.2019.2961339.

    Article  Google Scholar 

  37. Fioraldi A, D'Elia D C, Coppa E. WEIZZ: Automatic grey-box fuzzing for structured binary formats. In Proc. the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, July 2020, pp.1-13. https://doi.org/10.1145/3395363.3397372.

  38. Chen Y, Poskitt C M, Sun J et al. Learning-guided network fuzzing for testing cyber-physical system defences. In Proc. the 34th IEEE/ACM International Conference on Automated Software Engineering, November 2019, pp.962-973. https://doi.org/10.1109/ASE.2019.00093.

  39. Peng H, Shoshitaishvili Y, Payer M. T-Fuzz: Fuzzing by program transformation. In Proc. the 2018 IEEE Symposium on Security and Privacy, May 2018, pp.697-710. https://doi.org/10.1109/SP.2018.00056.

  40. Padhye R, Lemieux C, Sen K. JQF: Coverage-guided property-based testing in Java. In Proc. the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, July 2019, pp.398-401. https://doi.org/10.1145/3293882.3339002.

  41. Noller Y, Kersten R, Păsăreanu C S. Badger: Complexity analysis with fuzzing and symbolic execution. In Proc. the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, July 2018, pp.322-332. https://doi.org/10.1145/3213846.3213868.

  42. Zhou C, Wang M, Liang J et al. Zeror: Speed up fuzzing with coverage-sensitive tracing and scheduling. In Proc. the 35th IEEE/ACM International Conference on Automated Software Engineering, September 2020, pp.858-870. https://doi.org/10.1145/3324884.3416572.

  43. Wang T, Wei T, Gu G et al. TaintScope: A checksum-aware directed fuzzing tool for automatic software vulnerability detection. In Proc. the 2010 IEEE Symposium on Security and Privacy, May 2010, pp.497-512. https://doi.org/10.1109/SP.2010.37.

  44. Lemieux C, Sen K. Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage. In Proc. the 33rd ACM/IEEE International Conference on Automated Software Engineering, September 2018, pp.475-485. https://doi.org/10.1145/3238147.3238176.

  45. Situ L, Wang L, Li X, Guan L, Zhang W, Liu P. Energy distribution matters in greybox fuzzing. In Proc. the 41st IEEE/ACM International Conference on Software Engineering, May 2019, pp.270-271. https://doi.org/10.1109/ICSECompanion.2019.00109.

  46. Böhme M, Pham V T, Roychoudhury A. Coverage-based greybox fuzzing as Markov chain. IEEE Transactions on Software Engineering, 2017, 45(5): 489-506. https://doi.org/10.1109/TSE.2017.2785841.

  47. Wüstholz V, Christakis M. Harvey: A greybox fuzzer for smart contracts. In Proc. the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, November 2020, pp.1398-1409. https://doi.org/10.1145/3368089.3417064.

  48. Gan S, Zhang C, Qin X et al. CollAFL: Path sensitive fuzzing. In Proc. the 2018 IEEE Symposium on Security and Privacy, May 2018, pp.679-696. https://doi.org/10.1109/SP.2018.00040.

Download references

Author information

Authors and Affiliations

  1. Computer School, Beijing Information Science and Technology University, Beijing, 100101, China

    Jia-Ming Zhang, Zhan-Qi Cui, Huan-Huan Wu, Li-Wei Zheng & Jian-Bin Liu

  2. School of Information Science and Technology, Nantong University, Nantong, 226019, China

    Xiang Chen

Authors
  1. Jia-Ming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhan-Qi Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huan-Huan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Li-Wei Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jian-Bin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhan-Qi Cui.

Supplementary Information

ESM 1

(PDF 757 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, JM., Cui, ZQ., Chen, X. et al. DeltaFuzz: Historical Version Information Guided Fuzz Testing. J. Comput. Sci. Technol. 37, 29–49 (2022). https://doi.org/10.1007/s11390-021-1663-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1663-7

Keywords

Search

Navigation