Advanced Reinforcement Learning Algorithms to Optimize Design Verification
Authors: 
Zahra Aref, Rohit Suvarna, Bill Hughes, Sandeep Srinivasan, Narayan B. MandayamAuthors Info & Claims
Zahra Aref, Rohit Suvarna, Bill Hughes, Sandeep Srinivasan, Narayan B. MandayamAuthors Info & ClaimsArticle No.: 112, Pages 1 - 6
Abstract
Given the increasing complexity of integrated circuits, the utilization of machine learning in simulation-based hardware design verification (DV) has become crucial to ensure comprehensive coverage of hard-to-hit states. Our paper proposes a deep deterministic policy gradient (DDPG) algorithm combined with prioritized experience replay (PER) to determine the stimulus settings that result in the highest average FIFO depth in a modified exclusive shared invalid (MESI) cache controller architecture. This architecture includes four FIFOs, each corresponding to a distinct CPU. Through extensive experimentation, DDPG coupled with PER (DDPG-PER) proves to be more effective than DDPG with uniform experience replay in enhancing average FIFO depth and coverage within the DV process. Furthermore, our proposed DDPG-PER framework significantly increases the occurrence of higher FIFO depths, thereby addressing the challenges associated with reaching hard-to-hit states in DV. The proposed DDPG-PER and DDPG algorithms also demonstrate a larger average FIFO depth over four CPUs, requiring considerably less execution time than Bayesian Optimization (BO).
References
[1]
Hyojin Choi, In Huh, Seungju Kim, Jeonghoon Ko, Changwook Jeong, Hyeonsik Son, Kiwon Kwon, Joonwan Chai, Younsik Park, Jaehoon Jeong, et al. 2021. Application of Deep Reinforcement Learning to Dynamic Verification of DRAM Designs. In 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 523--528.
[2]
Alexandru Dinu and Petre Lucian Ogrutan. 2022. Reinforcement Learning Made Affordable for Hardware Verification Engineers. Micromachines 13, 11 (2022), 1887.
[3]
Saumil Gogri, Aakash Tyagi, Michael Quinn, and Jiang Hu. 2022. Transaction Level Stimulus Optimization in Functional Verification Using Machine Learning Predictors. In 2022 23rd International Symposium on Quality Electronic Design (ISQED). IEEE, 71--76.
[4]
Onur Guzey, Li-C Wang, Jeremy R Levitt, and Harry Foster. 2009. Increasing the efficiency of simulation-based functional verification through unsupervised support vector analysis. IEEE transactions on computer-aided design of integrated circuits and systems 29, 1 (2009), 138--148.
[5]
Matthew Hoffman, Eric Brochu, Nando De Freitas, et al. 2011. Portfolio Allocation for Bayesian Optimization. In UAI. 327--336.
[6]
William Hughes, Sandeep Srinivasan, Rohit Suvarna, and Maithilee Kulkarni. 2019. Optimizing design verification using machine learning: Doing better than random. arXiv preprint arXiv:1909.13168 (2019).
[7]
Donald R Jones. 2001. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization 21 (2001), 345--383.
[8]
Maithilee Rajendra Kulkarni et al. 2019. Improving coverage of simulation-based design verification using Machine Learning techniques. Master's thesis. The University of Texas at Austin.
[9]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
[10]
Ashok B Mehta. 2018. Constrained random verification (crv). In ASIC/SoC Functional Design Verification. Springer, 65--74.
[11]
Jonas Mockus. 1998. The application of Bayesian methods for seeking the extremum. Towards global optimization 2 (1998), 117.
[12]
Andreas Olofsson. 2017. Intelligent design of electronic assets (idea) & posh open source hardware (posh). Mountain View: DARPA (2017).
[13]
Mark S Papamarcos and Janak H Patel. 1984. A low-overhead coherence solution for multiprocessors with private cache memories. In Proceedings of the 11th annual international symposium on Computer architecture. 348--354.
[14]
Nícolas Pfeifer, Bruno V Zimpel, Gabriel AG Andrade, and Luiz CV dos Santos. 2020. A reinforcement learning approach to directed test generation for shared memory verification. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 538--543.
[15]
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015. Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015).
[16]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In International conference on machine learning. PMLR, 387--395.
[17]
Wilson Snyder. 2004. Verilator and systemperl. In North American SystemC Users' Group, Design Automation Conference.
[18]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[19]
Fanchao Wang, Hanbin Zhu, Pranjay Popli, Yao Xiao, Paul Bodgan, and Shahin Nazarian. 2018. Accelerating coverage directed test generation for functional verification: A neural network-based framework. In Proceedings of the 2018 on Great Lakes Symposium on VLSI. 207--212.
Index Terms
- Advanced Reinforcement Learning Algorithms to Optimize Design Verification
Index terms have been assigned to the content through auto-classification.
Recommendations
Comparison of end-to-end and hybrid deep reinforcement learning strategies for controlling cable-driven parallel robots
Highlights- This study develops and compares the end-to-end DDPG strategy and the hybrid DDPG strategy in controlling CDPRs.
AbstractDeep reinforcement learning (DRL) has been proven effective in learning policies of high-dimensional states and actions. Recently, a variety of robot manipulation tasks have been accomplished using end-to-end DRL strategies. An end-to-...
Deep Reinforcement Learning: From Q-Learning to Deep Q-Learning
Neural Information ProcessingAbstractAs the two hottest branches of machine learning, deep learning and reinforcement learning both play a vital role in the field of artificial intelligence. Combining deep learning with reinforcement learning, deep reinforcement learning is a method ...
Conversational Recommender System Using Deep Reinforcement Learning
RecSys '22: Proceedings of the 16th ACM Conference on Recommender SystemsDeep Reinforcement Learning (DRL) uses the best of both Reinforcement Learning and Deep Learning for solving problems which cannot be addressed by them individually. Deep Reinforcement Learning has been used widely for games, robotics etc. Limited work ...
Comments
Information & Contributors
Information
Published In
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
DAC '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,770 of 5,499 submissions, 32%
Upcoming Conference
DAC '25
- Sponsor:
- sigda
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 173Total Downloads
- Downloads (Last 12 months)173
- Downloads (Last 6 weeks)75
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in