skip to main content
research-article

Guiding Formal Verification Orchestration Using Machine Learning Methods

Published: 22 August 2018 Publication History

Abstract

Typical modern HW designs include many blocks associated with thousands of design properties. Having today's commercial formal verifiers utilize a complementary set of state-of-art formal algorithms is a key in enabling the formal verification tools to successfully cope with verification problems of different sizes, types, and complexities. Formal engines orchestration is the methodology used to pick the most appropriate formal engine for a specific verification problem. It assures proper scheduling of the formal engines to minimize the time consumed to solve individual design verification problems, hence highly impacts the time required to verify the overall design properties. This work proposes the utilization of supervised machine learning classification techniques to guide the orchestration step by predicting the formal engines that should be assigned to a design property. Up to 16,500 formal verification runs on RTL designs and their properties are used to train the classifier to create a prediction model. The classifier assigns any new verification problem to an appropriate list of formal engines associated with a probability distribution over the set of engines classes. Our results indicate how the proposed model is able to improve the formal suite total run-time by up to 59% of its maximum allowable time improvement using multi-classification-based orchestration and to nominate with 88% accuracy the appropriate formal engines for new-to-verify HW designs.

References

[1]
A. Sanghavi. 2010. What is formal verification?, eetasia.com. EE Times-Asia, 2010.
[2]
Y. Hoskote, P. Ho, and X. Zhao. Coverage estimation for symbolic model checking. In Proceedings of the 36th Annual ACM/IEEE Design Automation Conference (DAC’99), 300--305.
[3]
A. P. Sistla and E. M. Clarke. 1985. The complexity of propositional linear temporal logics. ACM Journal of the Association for Computing Machinery 32, 3, 733--749.
[4]
J. R. Burch. E. M. Clarke, and K. L. McMillan. 1990. Symbolic model checking: 1020 States and beyond. School of Computer Science, Carnegie Mellon University.
[5]
J. R. Burch, E. M. Clarke, D. E. Long, K. L. McMillan, and D. L. Dill. 1994. Symbolic model checking for sequential circuit verification. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 1994, 401--424.
[6]
E. Goldberg and Y. Novikov. 2003. Verification of proofs of un-satisfiability for CNF formulas. In Proceedings of the Design Automation and Test in Europe (DATE). 886--891.
[7]
J. P. Marques-Silva and K. A. Sakallah. 1999. GRASP: A search algorithm for propositional satisfiability. IEEE Transactions on Computers, 1999, 506--521.
[8]
M. Sheeran, S. Singh, and G. Stalmarck. 2000. Checking safety properties using induction and a SAT-solver. In Proceedings of the 3rd International Conference on Formal Methods in Computer Aided Design (FMCAD). 108--125.
[9]
R. Bradley. 2011. SAT-Based model checking without unrolling. In Proceedings of 12th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI). 70--78.
[10]
L. C. Wang and F. Somenzi. 2003. A satisfiability based approach to abstraction refinement in model checking. In Proceedings of the 1st International Workshop on Bounded Model Checking (BMC).
[11]
Questa Formal Property Check. Retrieved from https://www.mentor.com/products/fv/questa-property-checking.
[12]
R. Jain, S. Mitra, J. Baumgartner, and P. Dasgupta. 2007. Formal assertion based verification in industrial setting. Tutorial in Design Automation Conference (DAC).
[13]
Symbolic Model Checking. Retrieved from http://www.springer.com/cda/content/document/cda_downloaddocument/9780387341552-c2.pdf?SGWID=0-0-45-321359-p173660384.
[14]
R. E. Bryant. 1986. Graph-based algorithms for Boolean function manipulation. IEEE Trans. Comp, C-35, Aug. 1986.
[15]
M. R. Prasad, A. Biere, and A. Gupta. 2005. A survey of recent advances in SAT-based formal verification. The International Journal on Software Tools for Technology. 156--173.
[16]
A. Biere, A. Cimatti, E. Clarke, and Y. Zhu. 1999. Symbolic model checking without BDDs. In Proceedings of the 23rd International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). 193--207.
[17]
E. M. Clarke, A. Gupta, J. Kukula, and O. Strichman. 2002. SAT-based abstraction refinement using ILP and machine learning techniques. In Proceedings of the International Conference on Computer Aided Verification (CAV). 265--279.
[18]
J. Rice. 1976. The algorithm selection problem. Journal of Advances in Computers, 1976, 15, 65--118
[19]
Algorithm Selection Literature Summary. 2018. https://larskotthoff.github.io/assurvey/.
[20]
L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown. 2008. SATzilla: Portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 2008, 32, 565--606.
[21]
S. Kadioglu, Y. Malitsky, A. Sabharwal, H. Samulowitz, and M. Sellmann. 2011. Algorithm selection and scheduling. In Proceedings of the International Conference on Principles and Practice of Constraint Programming, 2011, 454--469.
[22]
R. Amadini, M. Gabbrielli, and J. Mauro. 2014. SUNNY: A lazy portfolio approach for constraint solving. Theory and Practice of Logic Programming, 2014, 509--524.
[23]
L. Kottho. 2014. Algorithm selection for combinatorial search problems: A survey. AI Magazine, 2014, 35, 48--60.
[24]
B. Bischl, P. Kerschke, L. Kottho, M. Lindauer, Y. Malitsky, A. Frechette, H. Hoos, F. Hutter, K. Leyton-Brown, K. Tierney, and J. Vanschoren. 2016. ASlib: A benchmark library for algorithm selection. Artificial Intelligence Journal, 2016, 41--58.
[25]
K. A. Smith-Miles. 2009. Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computer Survey, 2009, 41, 1--25.
[26]
M. Maratea, L. Pulina, and F. Ricca. 2012. Applying Machine Learning Techniques to ASP Solving. University of Sassari Tech. Rep, 37--48.
[27]
L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown. 2011. Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In Proceedings of the International Joint Conference on Artificial Intelligence.
[28]
M. Gagliolo and J. Schmidhuber. 2007. Learning dynamic algorithm portfolios. Annals of Mathematics and Artificial Intelligence, 47, 295--328.
[29]
H. Hoos, R. Kaminski, M Lindauer, and T. Schaub. 2015. ASPEED: Solver scheduling via answer set programming. Theory and Practice of Logic Programming, 15, 117--142.
[30]
T. Carchrae and T. J. C. Beck. 2004. Low-knowledge algorithm control. In Proceedings of the National Conference on Artificial Intelligence, 2004, 4954.
[31]
A. Guerri and M. Milano. 2004. Learning techniques for automatic algorithm portfolio selection. In Proceedings of the Conference on Artificial Intelligence. 475--479.
[32]
E. Nudelman, K. Leyton-Brown, A. Devkar, Y. Shoham, and H. Hoos. 2004. Understanding random SAT: Beyond the clauses-to-variables ratio. In Proceedings of the Conference on Principles and Practice of Constraint Programming. 438--452.
[33]
M. Lindauer, H. Hoos, and F. Hutter. 2015. From sequential algorithm selection to parallel portfolio selection. In Proceedings of the International Conference on Learning and Intelligent Optimization. 1--16.
[34]
H. Hoos, M. Lindauer, and T. Schaub. 2014. Claspfolio 2: Advances in algorithm selection for answer set programming. Theory and Practice of Logic Programming, 14, 569--585.
[35]
E. El Mandouh and A. G. Wassal. 2016. Estimation of formal verification cost using regression machine learning. In Proceedings of the IEEE International High Level Design Validation and Test Workshop (HLDVT). 121--127
[36]
Incisive Formal Verifier. Retrieved from https://community.cadence.com//CSSharedFiles/forums/storage/22/10078/IncisiveFV_ds.pdf.
[37]
J. Hogan. 2016. 9 major and 23 minor Formal ABV tool metrics -- plus their gotchas. Deep Chip. Retrieved from http://www.deepchip.com/items/0558-03.html.
[38]
M. Handover. 2015. Back to Basics -- Doing Formal the Right Way. Retrieved from http://www.techdesignforums.com/practice/technique/four-steps-formal-verification-test-plan.
[39]
J. Yu. 2013. Formal verification results prediction, 2013, Patent US20140330758. Retrieved from https://patents.justia.com/patent/20140330758.
[40]
R. Ho, M. Theoblad, M. Deneroff, R. O. Dror, J. Gagliardo, and D. Shaw. 2008. Early formal verification of conditional coverage points to identify intrinsically hard-to-verify logic. In Proceedings of 45th ACM/IEEE Design Automation Conference (DAC). 268--271.
[41]
E. Fox and C. Guestrin. 2015. Machine learning: Classification, University of Washington, Coursera. Retrieved from https://www.coursera.org/learn/ml-classification/home/welcome.
[42]
Y. Bengio. 2012. Practical Recommendations for Gradient-Based Training of Deep Architectures. Neural Networks: Tricks of the Trade. Springer, Berlin, 2012. 437--478.
[43]
GraphLib Create Package. Retrieved from https://github.com/dato-code/SFrame.
[44]
J. Zhu, H. Zou, S. Rosset, and T. Hastie. 2009. Multi-class adaBoost. Statistics and Its Interface, 2009, 2, 349--360
[45]
L. Held and D. S. Bove. 2014. Applied statistical inference. Springer-Verlag, Berlin, 13--50.
[46]
J. Friedman and B. Popescu. 2014. Gradient-Directed Regularization for Linear Regression and Classification. Department of Statistics and Stanford Linear Accelerator Center, Stanford University, 1--40.
[47]
AIGER. http://fmv.jku.at/aiger.
[48]
E. El-Mandouh and A. G. Wassal. 2010. Automatic generation of hardware design properties from simulation traces. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), 2317--2320, 2010.
[49]
R. Kohavi and G. John. 1997. Wrappers for feature subset selection. Artificial Intelligence Journal, 15, 273--324.
[50]
A. Jain. 2016. A Complete Tutorial on Ridge and Lasso Regression in Python. Retrieved from http://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python.
[51]
J. Brownlee. 2014. Classification Accuracy is Not Enough: More Performance Measures You Can Use. Retrieved from http://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/.
[52]
L. Sachan. 2015. Logistic regression vs decision trees vs SVM. Retriecved from https://www.edvancer.in/logistic-regression-vs-decision-trees-vs-svm-part1.
[53]
S. Arlot and A. Celisse. 2010. A survey of cross-validation procedures for model selection. Statistics Surveys, 2010, 40--79.
[54]
A. Ng, J. Ngiam, C. Yu Foo, Y. Mai, and C. Suen. 2013. Stanford deep learning tutorial UFLDL, Softmax Regression, 2013. http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/.
[55]
L. Kottho. 2013. LLAMA: Leveraging learning to automatically manage algorithms. Tech. Rep arXiv:1306.1031. arXiv, 2013. http://arxiv.org/abs/1306.1031.
[56]
M. Lindauer, J. N. van Rijn, and L. Kottho. 2017. Open algorithm selection challenge 2017 setup and scenarios. In Proceedings of Machine Learning Research, 2017. 1--7

Cited By

View all
  • (2024)Survey of Machine Learning for Software-assisted Hardware Design Verification: Past, Present, and ProspectACM Transactions on Design Automation of Electronic Systems10.1145/366130829:4(1-42)Online publication date: 24-Apr-2024
  • (2024)Robust battery lifetime prediction with noisy measurements via total-least-squares regressionIntegration, the VLSI Journal10.1016/j.vlsi.2023.10213696:COnline publication date: 1-May-2024
  • (2022)Analyzing and Predicting Verification of Data-Aware Process Models–A Case Study With Spectrum AuctionsIEEE Access10.1109/ACCESS.2022.315444510(31699-31713)Online publication date: 2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Design Automation of Electronic Systems
ACM Transactions on Design Automation of Electronic Systems  Volume 23, Issue 5
September 2018
310 pages
ISSN:1084-4309
EISSN:1557-7309
DOI:10.1145/3268934
  • Editor:
  • Naehyuck Chang
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

Publication History

Published: 22 August 2018
Accepted: 01 May 2018
Revised: 01 May 2018
Received: 01 December 2017
Published in TODAES Volume 23, Issue 5

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Formal verification
  2. algorithm selection
  3. classification ML

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)37
  • Downloads (Last 6 weeks)0
Reflects downloads up to 23 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Survey of Machine Learning for Software-assisted Hardware Design Verification: Past, Present, and ProspectACM Transactions on Design Automation of Electronic Systems10.1145/366130829:4(1-42)Online publication date: 24-Apr-2024
  • (2024)Robust battery lifetime prediction with noisy measurements via total-least-squares regressionIntegration, the VLSI Journal10.1016/j.vlsi.2023.10213696:COnline publication date: 1-May-2024
  • (2022)Analyzing and Predicting Verification of Data-Aware Process Models–A Case Study With Spectrum AuctionsIEEE Access10.1109/ACCESS.2022.315444510(31699-31713)Online publication date: 2022
  • (2021)State-of-the-art OCR Algorithm for Translation Guiding: An Online-offline Combinational Model2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA51532.2021.9544563(213-217)Online publication date: 2-Sep-2021

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media