Sarah Amir
Domenic Forte
ABSTRACT
Benchmarking can drive the development of technologies by facilitating standardization of features for comparison of different methods. While hardware security has seen an exponential growth in innovation throughout the last decade, the lack of sufficient benchmarks for data-driven analysis is prominent. Researchers must currently rely on decades-old VLSI benchmarks, which in most cases were not designed with security evaluation in mind. Considering the present day computational power, these benchmarks lack in both quality and quantity for usage in hardware security topics such as obfuscation and hardware Trojans. Many advanced techniques, like statistical analysis and machine learning, require a large number of samples in order to sufficiently examine the feature space. In an attempt to resolve this issue, we have developed the first synthetic benchmark generation process flow. This paper describes our novel technique that utilizes linear optimization to generate an endless number of synthetic combinational benchmarks that are adaptable to user input constraints and divergent in quantifiable structural features from input reference benchmarks. Thus, our framework offers customization for generating richer and more challenging benchmarks for data-driven hardware security. Through experimentation, we verify that our benchmarks offers more structural variation than the current benchmark suites.
- Christoph Albrecht. 2005. IWLS 2005 benchmarks. In International Workshop for Logic Synthesis (IWLS). 9.Google Scholar
- Luca Amarú, Pierre-Emmanuel Gaillardon, and Giovanni De Micheli. 2015. The EPFL combinational benchmark suite. In Proceedings of the 24th International Workshop on Logic & Synthesis (IWLS).Google Scholar
- Sarah Amir, Bicky Shakya, Xiaolin Xu, Yier Jin, Swarup Bhunia, Mark Tehranipoor, and Domenic Forte. 2018. Development and evaluation of hardware obfuscation benchmarks. Journal of Hardware and Systems Security 2, 2 (2018), 142--161.Google ScholarCross Ref
- Johanna Baehr, Alessandro Bernardini, Georg Sigl, and Ulf Schlichtmann. 2020. Machine learning and structural characteristics for reverse engineering. Integration 72 (2020), 1--12.Google ScholarDigital Library
- Berkeley Logic Synthesis and Verification Group. 2004. ABC: A System for Sequential Synthesis and Verification. http://www.eecs.berkeley.edu/~alanmi/abc/Google Scholar
- F. Brglez, D. Bryan, and K. Kozminski. 1989. Combinational profiles of sequential benchmark circuits. In Circuits and Systems, 1989., IEEE International Symposium on. 1929--1934 vol.3. Google ScholarCross Ref
- F. Brglez and H. Fujiwara. 1985. A Neutral Netlist of 10 Combinational Benchmark Circuits and a Target Translator in Fortran. In Proceedings of IEEE Int'l Symposium Circuits and Systems (ISCAS 85). IEEE Press, Piscataway, N.J., 677--692.Google Scholar
- F. Corno, M.S. Reorda, and G. Squillero. 2000. RT-level ITC'99 benchmarks and first ATPG results. Design Test of Computers, IEEE 17, 3 (Jul 2000), 44--53. Google ScholarDigital Library
- Rana Elnaggar and Krishnendu Chakrabarty. 2018. Machine learning for hardware security: opportunities and risks. Journal of Electronic Testing 34, 2 (2018), 183--201.Google ScholarDigital Library
- L Goldstein. 1979. Controllability/observability analysis of digital circuits. IEEE Transactions on Circuits and Systems 26, 9 (1979), 685--693.Google ScholarCross Ref
- Matthew R Guthaus, Jeffrey S Ringenberg, Dan Ernst, Todd M Austin, Trevor Mudge, and Richard B Brown. 2001. MiBench: A free, commercially representative embedded benchmark suite. In Proceedings of the fourth annual IEEE international workshop on workload characterization. WWC-4 (Cat. No. 01EX538). IEEE, 3--14.Google ScholarCross Ref
- Michael Hutton, Jerry P Grossman, Jonathan Rose, and Derek Corneil. 1996. Characterization and parameterized random generation of digital circuits. In Proceedings of the 33rd annual Design Automation Conference. ACM, 94--99.Google ScholarDigital Library
- Michael Hutton, Jonathan Rose, and Derek Corneil. 2003. Clustered And Iterative Synthetic Circuit Generation. http://www.eecg.toronto.edu/~jayar/software/Cgen/Cgen.htmlGoogle Scholar
- Michael D Hutton, Jonathan Rose, Jerry P Grossman, and Derek G Corneil. 1998. Characterization and parameterized generation of synthetic combinational benchmark circuits. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 17, 10 (1998), 985--996.Google ScholarDigital Library
- Nevin Kapur, Debabrata Ghosh, and Franc Brglez. 1997. Towards a new benchmarking paradigm in EDA: analysis of equivalence class mutant circuit distributions. In Proceedings of the 1997 international symposium on Physical design. 136--143.Google ScholarDigital Library
- Krzysztof Koźmiński. 1991. Benchmarks for layout synthesis---evolution and current status. In Proceedings of the 28th ACM/IEEE Design Automation Conference. 265--270.Google ScholarDigital Library
- Martin Kutter and Fabien AP Petitcolas. 1999. Fair benchmark for image watermarking systems. In Security and Watermarking of Multimedia Contents, Vol. 3657. International Society for Optics and Photonics, 226--239.Google ScholarCross Ref
- Bernard S Landman and Roy L Russo. 1971. On a pin versus block relationship for partitions of logic graphs. IEEE Transactions on computers 100, 12 (1971), 1469--1479.Google ScholarDigital Library
- Chunho Lee, Miodrag Potkonjak, and William H Mangione-Smith. 1997. Mediabench: A tool for evaluating and synthesizing multimedia and communications systems. In Proceedings of 30th Annual International Symposium on Microarchitecture. IEEE, 330--335.Google Scholar
- David G Luenberger, Yinyu Ye, et al. 1984. Linear and nonlinear programming. Vol. 2. Springer.Google Scholar
- Synopsys User Manual. 2005. TetraMAX ATPG user guide. Version X-2005.09 (2005), 249--264.Google Scholar
- Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621 (2017).Google Scholar
- Hassan Salmani, Mohammad Tehranipoor, and Ramesh Karri. 2013. On design vulnerability analysis and trust benchmarks development. In 2013 IEEE 31st international conference on computer design (ICCD). IEEE, 471--474.Google ScholarCross Ref
- Bicky Shakya, Tony He, Hassan Salmani, Domenic Forte, Swarup Bhunia, and Mark Tehranipoor. 2017. Benchmarking of Hardware Trojans and Maliciously Affected Circuits. Journal of Hardware and Systems Security 1, 1 (2017), 85--102.Google ScholarCross Ref
- Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data 6, 1 (2019), 60.Google ScholarCross Ref
- Dirk Stroobandt, Peter Verplaetse, and Jan Van Campenhout. 2000. Generating synthetic benchmark circuits for evaluating CAD tools. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 19, 9 (2000), 1011--1022.Google ScholarDigital Library
- Pramod Subramanyan, Sayak Ray, and Sharad Malik. 2015. Evaluating the security of logic encryption algorithms. In IEEE Intl. Symposium on HOST 2015, Washington, DC, USA, 2015. 137--143.Google ScholarCross Ref
- Pramod Subramanyan, Sayak Ray, and Sharad Malik. 2015. SAT Attack tool. https://bitbucket.org/spramod/host15-logic-encryptionGoogle Scholar
- Benjamin Tan, Ramesh Karri, Nimisha Limaye, Abhrajit Sengupta, Ozgur Sinanoglu, Md Moshiur Rahman, Swarup Bhunia, Danielle Duvalsaint, Amin Rezaei, Yuanqi Shen, et al. 2020. Benchmarking at the Frontier of Hardware Security: Lessons from Logic Locking. arXiv preprint arXiv:2006.06806 (2020).Google Scholar
- Tao Wan and M. Chrzanowska-Jeske. 2004. Generating random benchmark circuits for floorplanning. In 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512), Vol. 5. V--V.Google Scholar
- Herwig Van Marck, Dirk Stroobandt, and Jan Van Campenhout. 1995. Towards an extension of Rent's rule for describing local variations in interconnection complexity. In Proc. 4th Intl. Conf. for Young Computer Scientists. 136--141.Google Scholar
- Peter Verplaetse, Jan Van Campenhout, and Dirk Stroobandt. 2000. On synthetic benchmark generation methods. In 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No. 00CH36353), Vol. 4. IEEE, 213--216.Google Scholar
- Sheng Wei, Kai Li, Farinaz Koushanfar, and Miodrag Potkonjak. 2012. Hardware Trojan horse benchmark via optimal creation and placement of malicious circuitry. In Proceedings of the 49th Annual Design Automation Conference. 90--95.Google ScholarDigital Library
- Muhammad Yasin, Jeyavijayan J. V. Rajendran, Ozgur Sinanoglu, and Ramesh Karri. 2016. On Improving the Security of Logic Locking. IEEE Trans. on CAD of Integrated Circuits and Systems 35, 9 (2016), 1411--1424.Google ScholarDigital Library
Index Terms
- Adaptable and divergent synthetic benchmark generation for hardware security
Recommendations
Requirements-driven database systems benchmark method
Benchmarks are the vital tools in the performance measurement, evaluation, and comparison of relational database management systems (RDBMS). Standard benchmarks such as the TP1, TPC-A, TPC-B, TPC-C, TPC-D, TPC-H, TPC-R, TPC-W, Wisconsin, and AS3 Ap ...
Using software architectural patterns for synthetic embedded multicore benchmark development
IISWC '12: Proceedings of the 2012 IEEE International Symposium on Workload Characterization (IISWC)Benchmarks capture the essence of many important real-world applications and allow performance, and power analysis while developing new systems. Synthetic benchmarks are a miniaturized form of benchmarks that allow high simulation speeds and act as ...
Subsetting the SPEC CPU2006 benchmark suite
On August 24, 2006, the Standard Performance Evaluation Corporation (SPEC) announced CPU2006 -- the next generation of industry-standardized CPU-intensive benchmark suite. The SPEC CPU benchmark suite has become the most frequently used suite for ...
Comments