Exploring Coverage Metrics in Hardware Fuzzing: A Comprehensive Analysis
Pages 240 - 245
Abstract
The increasing complexity and integration of diverse components in modern System-on-Chip (SoC) designs make them susceptible to a range of attacks. Unfortunately, a substantial disjunction persists between the sophisticated architectures of the SoCs and Design Verification (DV) techniques to detect such vulnerabilities. Recently, Hardware fuzzing, inspired by software testing, has been gaining attention for its efficient bug-detection capabilities in SoC designs. Coverage metrics serve as a pivotal tool in assessing the efficacy of fuzzing techniques by gauging the extent to which the Design Under Test (DUT) design space is explored during the verification process. This paper endeavors to delve into various hardware coverage metrics, encompassing branch, statement, Finite State Machine (FSM), line, and expression coverage, in order to elucidate both the merits and demerits of existing hardware fuzzing methodologies. Furthermore, it seeks to explore how these coverage metrics can be harnessed to bolster the efficacy of hardware fuzzing, thereby augmenting bug detection rates and streamlining testing endeavors. This work provides an analysis on different coverage metrics that could be utilized and the impact of it on the overall design coverage for various IP blocks and CPU designs.
References
[1]
Armaiti Ardeshiricham, Wei Hu, and Ryan Kastner. 2017. Clepsydra: Modeling timing flows in hardware designs. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD).
[2]
Nitay Artenstein. 2017. BroadPWN: Remotely Compromising Android and iOS Via a Bug in Broachom’s Wi-Fi Chipsets. In BlackHat USA.
[3]
Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and Raoul Strackx. 2018. Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution. In USENIX Security Symposium.
[4]
Cadence. [n. d.]. Cadence Webpage. https://www.cadence.com/en_US/home.html Last accessed: 04/29/2024.
[5]
Cadence Inc. [n. d.]. JasperGold Formal Verification Platform. https://www.cadence.com/en US/home/tools/system-design-and-verification/formal-and-static-verification/jasper-gold-verification-platform.html Last accessed: 04/29/2024.
[6]
S. Canakci, C. Rajapaksha, L. Delshadtehrani, A. Nataraja, M. Taylor, M. Egele, and A. Joshi. 2023. ProcessorFuzz: Processor Fuzzing with Control and Status Registers Guidance. In IEEE International Symposium on Hardware Oriented Security and Trust (HOST).
[7]
Mingsong Chen and Prabhat Mishra. 2011. Property Learning Techniques for Efficient Generation of Directed Tests. IEEE Trans. Comput. 60, 6 (Feb 2011), 852–864.
[8]
Cisco. [n. d.]. CVE-2021-34696 in Cisco Routers. https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-asr900acl-UeEyCxkv Last accessed: 04/29/2024.
[9]
Ghada Dessouky, David Gens, Patrick Haney, Garrett Persyn, Arun Kanuparthi, Hareesh Khattri, Jason M. Fung, Ahmad-Reza Sadeghi, and Jeyavijayan Rajendran. 2019. Hardfails: Insights into Software-Exploitable Hardware Bugs. In USENIX Conference on Security Symposium.
[10]
Sai Manoj P Dinakarrao, H. Yu, C. Gu, and C. Zhuo. 2014. A zonotoped macromodeling for reachability verification of eye-diagram in high-speed I/O links with jitter. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD).
[11]
Nusrat Farzana, Fahim Rahman, Mark Tehranipoor, and Farimah Farahmandi. 2019. SoC Security Verification using Property Checking. In IEEE International Test Conference (ITC).
[12]
Weimin Fu, Orlando Arias, Yier Jin, and Xiaolong Guo. 2021. Fuzzing Hardware: Faith or Reality? : Invited Paper. In IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).
[13]
Saumil Gogri, Aakash Tyagi, Michael Quinn, and Jiang Hu. 2022. Transaction Level Stimulus Optimization in Functional Verification Using Machine Learning Predictors. In International Symposium on Quality Electronic Design (ISQED).
[14]
Google. [n. d.]. Americal Fuzzy Loop. https://github.com/google/AFL Last accessed: 04/29/2024.
[15]
Onur Guzey, Li-C. Wang, Jeremy R. Levitt, and Harry Foster. 2010. Increasing the Efficiency of Simulation-Based Functional Verification Through Unsupervised Support Vector Analysis. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 29, 1 (2010), 138–148.
[16]
Muhammad Monir Hossain, Arash Vafaei, Kimia Zamiri Azar, Fahim Rahman, Farimah Farahmandi, and Mark Tehranipoor. 2023. SoCFuzzer: SoC Vulnerability Detection using Cost Function enabled Fuzz Testing. In Design, Automation & Test in Europe Conference & Exhibition (DATE).
[17]
Wei Hu, Armaiti Ardeshiricham, Mustafa S Gobulukoglu, Xinmu Wang, and Ryan Kastner. 2018. Property Specific Information Flow Analysis for Hardware Security Verification. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD).
[18]
Jaewon Hur, Suhwan Song, Dongup Kwon, Eunjin Baek, Jangwoo Kim, and Byoungyoung Lee. 2021. DifuzzRTL: Differential Fuzz Testing to Find CPU Bugs. In IEEE Symposium on Security and Privacy (SP).
[19]
Intel. 2019. Machine Check Error Avoidance on Page Size Change/CVE-2018-12207. https://www.intel.com/content/www/us/en/developer/articles/troubleshooting/software-security-guidance/advisory-guidance/machine-check-error-avoidance-page-size-change.html Last accessed: 04/29/2024.
[20]
Nursultan Kabylkas, Tommy Thorn, Shreesha Srinath, Polychronis Xekalakis, and Jose Renau. 2021. Effective Processor Verification with Logic Fuzzer Enhanced Co-Simulation. In IEEE/ACM International Symposium on Microarchitecture.
[21]
Rahul Kande, Addison Crump, Garrett Persyn, Patrick Jauernig, Ahmad-Reza Sadeghi, Aakash Tyagi, and Jeyavijayan Rajendran. 2022. TheHuzz: Instruction Fuzzing of Processors Using Golden-Reference Models for Finding Software-Exploitable Vulnerabilities. In USENIX Security Symposium (USENIX Security).
[22]
Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, and Yuval Yarom. 2019. Spectre Attacks: Exploiting Speculative Execution. In IEEE Symposium on Security and Privacy (S&P’19).
[23]
Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan Bachrach, and Koushik Sen. 2018. RFUZZ: Coverage-Directed Fuzz Testing of RTL on FPGAs. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD).
[24]
T Li, H Zou, Luo D, and Qu W. 2021. Symbolic simulation enhanced coverage-directed fuzz testing of RTL design. In IEEE International Symposium on Circuits and Systems (ISCAS).
[25]
J. Lin, S. Zhu, Z. Yu, D. Xu, Sai Manoj Pudukotai Dinakarrao, and H. Yu. 2015. A scalable and reconfigurable 2.5D integrated multicore processor on silicon interposer. In IEEE Custom Integrated Circuits Conf.
[26]
Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Anders Fogh, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, Yuval Yarom, and Mike Hamburg. 2018. Meltdown: Reading Kernel Memory from User Space. In USENIX Security Symposium.
[27]
Yu Liu, Yier Jin, Aria Nosratinia, and Yiorgos Makris. 2017. Silicon Demonstration of Hardware Trojan Design and Detection in Wireless Cryptographic ICs. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 25, 4 (2017), 1506–1519.
[28]
Valentin J. M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2018. Fuzzing: Art, Science, and Engineering. CoRR abs/1812.00140 (2018).
[29]
MITRE Corportation. [n. d.]. CVE-2023-29856 in D-Link. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29856 Last accessed: 04/29/2024.
[30]
Sujit Kumar Muduli, Gourav Takhar, and Pramod Subramanyan. 2020. HyperFuzzing for SoC Security Validation. In IEEE/ACM International Conference On Computer Aided Design (ICCAD).
[31]
Rajdeep Mukherjee, Daniel Kroening, and Tom Melham. 2015. Hardware Verification Using Software Analyzers. In IEEE Computer Society Annual Symposium on VLSI.
[32]
L. Ni, Sai Manoj P Dinakarrao, Y. Song, C. Gu, and H. Yu. 2016. A Zonotoped Macromodeling for Eye-Diagram Verification of High-Speed I/O Links With Jitter and Parameter Variations. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems 35, 6 (Jun. 2016), 1040–1051.
[33]
NIST National Vulnerability Database - HP. [n. d.]. CVE-2004-2439 in HP Printers. https://nvd.nist.gov/vuln/detail/CVE-2004-2439 Last accessed: 04/29/2024.
[34]
NIST National Vulnerability Database - Microsoft. [n. d.]. CVE-2004-2439 in Microsoft Xbox. https://nvd.nist.gov/vuln/detail/CVE-2007-1221 Last accessed: 04/29/2024.
[35]
NVIDIA. [n. d.]. CVE-2021-1088 in NVIDIA GPU and Tegra Hardware. https://nvidia.custhelp.com/app/answers/detail/a_id/5263 Last accessed: 04/29/2024.
[36]
NVIDIA Corportation. [n. d.]. CVE-2021-1071 in NVIDIA Hardware. https://nvidia.custhelp.com/app/answers/detail/a_id/5147 Last accessed: 04/29/2024.
[37]
Andreas Olofsson. [n. d.]. Intelligent Design of Electronic Assets (IDEA) & Posh Open Source Hardware (POSH). https://www.darpa.mil/attachments/eri_design_proposers_day.pdf Last accessed: 04/29/2024.
[38]
Maoyuan Qin, Xinmu Wang, Baolei Mao, Dejun Mu, and Wei Hu. 2020. A Formal Model for Proving Hardware Timing Properties and Identifying Timing Channels. Integr. VLSI J. 72, C (May 2020), 123–133.
[39]
Smruti R. Sarangi, Abhishek Tiwari, and Josep Torrellas. 2006. Phoenix: Detecting and Recovering from Permanent Processor Design Bugs with Programmable Hardware. In IEEE/ACM International Symposium on Microarchitecture.
[40]
Raghul Saravanan and Sai Manoj Pudukotai Dinakarrao. 2024. The Emergence of Hardware Fuzzing: A Critical Review of its Significance. arxiv:2403.12812 [cs.CR]
[41]
Michael Schwarz, Moritz Lipp, Daniel Moghimi, Jo Van Bulck, Julian Stecklina, Thomas Prescher, and Daniel Gruss. 2019. ZombieLoad: Cross-Privilege-Boundary Data Sampling. In ACM SIGSAC Conference on Computer and Communications Security.
[42]
Kostya Serebryany. 2017. OSS-Fuzz - Google’s continuous fuzzing service for open source software. In USENIX Security Symposium.
[43]
Y. Song, Sai Manoj P Dinakarrao, and H. Yu. 2014. Zonotope-based nonlinear model order reduction for fast performance bound analysis of analog circuits with multiple-interval-valued parameter variations. In Design, Automation Test in Europe Conference Exhibition (DATE).
[44]
Y. Song, H. Yu, and Sai Manoj P Dinakarrao. 2014. Reachability-Based Robustness Verification and Optimization of SRAM Dynamic Stability Under Process Variations. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems 33, 4 (Apr. 2014), 585–598.
[45]
Michael Sutton, Adam Greene, and Pedram Amini. 2007. Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley Professional.
[46]
Shibo Tang, Xingxin Wang, Yifei Gao, and Wei Hu. 2022. Accelerating SoC Security Verification and Vulnerability Detection Through Symbolic Execution. In International SoC Design Conference (ISOCC).
[47]
Timothy Trippel, Kang G. Shin, Alex Chernyakhovsky, Garret Kelly, Dominic Rizzo, and Matthew Hicks. 2022. Fuzzing Hardware Like Software. In USENIX Security Symposium (USENIX Security).
[48]
Verilator. [n. d.]. Welcome to Verilator. https://www.veripool.org/verilator/ Last accessed: 04/29/2024.
[49]
Ilya Wagner and Valeria Bertacco. 2007. Engineering Trust with Semantic Guardians. In Design, Automation & Test in Europe Conference & Exhibition.
[50]
Fanchao Wang, Hanbin Zhu, Pranjay Popli, Yao Xiao, Paul Bodgan, and Shahin Nazarian. 2018. Accelerating Coverage Directed Test Generation for Functional Verification: A Neural Network-Based Framework. In Great Lakes Symposium on VLSI.
[51]
Jin Yang and A. Puder. 2005. Tightly integrate dynamic verification with formal verification: a GSTE based approach. In Asia and South Pacific Design Automation Conference.
[52]
Song Yang, Sai Manoj P Dinakarrao, and Hao Yu. 2014. A robustness optimization of SRAM dynamic stability by sensitivity-based reachability analysis. In Asia and South Pacific Design Automation Conference (ASP-DAC).
[53]
Danfeng Zhang, Yao Wang, G. Edward Suh, and Andrew C. Myers. 2015. A Hardware Design Language for Timing-Sensitive Information-Flow Security. In International Conference on Architectural Support for Programming Languages and Operating Systems.
Recommendations
On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs
ICSTW '13: Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation WorkshopsCombinatorial test suite design is a test generation technique, popular in part due to its ability to achieve coverage and defect finding power approximating that of exhaustive testing while keeping test suite sizes constrained. In recent years, there ...
Coverage metrics for formal verification
In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is and whether it really covers all the behaviors of ...
Comments
Information & Contributors
Information
Published In
June 2024
797 pages
ISBN:9798400706059
DOI:10.1145/3649476
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Commonwealth Cyber Initiative
Conference
GLSVLSI '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 312 of 1,156 submissions, 27%
Upcoming Conference
GLSVLSI '25
- Sponsor:
- sigda
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 137Total Downloads
- Downloads (Last 12 months)137
- Downloads (Last 6 weeks)15
Reflects downloads up to 25 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format