skip to main content
10.1145/3649329.3657335acmconferencesArticle/Chapter ViewAbstractPublication PagesdacConference Proceedingsconference-collections
research-article

Using Probabilistic Model Rollouts to Boost the Sample Efficiency of Reinforcement Learning for Automated Analog Circuit Sizing

Published: 07 November 2024 Publication History

Abstract

Despite recent advances in algorithms, such as the use of reinforcement learning, analog circuit sizing optimization remains a challenging task that demands numerous circuit simulations, hence extensive CPU times. This paper introduces the application of Model-Based Policy Optimization (MBPO) to highly boost the sample efficiency of reinforcement learning for analog circuit sizing. This method leverages an ensemble of probabilistic dynamic models to generate short rollouts branched from real data for a fast but extensive exploration of the design space, thereby speeding up the learning process of the reinforcement learning agent and improving its convergence. Integrated in the Twin Delayed DDPG (TD3) algorithm, our new model-based TD3 (MBTD3) approach is validated on analog circuits of different complexity, outperforming the existing model-free TD3 method by achieving power/area-optimal design solutions within up to ~3x fewer simulations and half the run time. In addition, for larger analog circuits, we present a multi-agent version of MBTD3, in which multiple simultaneous agents use global probabilistic models for sizing the different sub-blocks within the circuit. Demonstrated for a complex data receiver circuit, it surpasses the model-free multi-agent TD3 method with ~2x less simulations and half the run time. The proposed novel algorithms clearly boost the efficiency of automated analog circuit sizing.

References

[1]
Manuel Barros et al. Analog Circuits and Systems Optimization based on Evolutionary Computation Techniques, volume 294 of Studies in Computational Intelligence. Springer Berlin Heidelberg.
[2]
Georges Gielen et al. Computer-aided design of analog and mixed-signal integrated circuits. Proceedings of the IEEE, 88(12):1825--1854, 2000.
[3]
Glenn Wolfe et al. Extraction and use of neural network models in automated synthesis of operational amplifiers. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 22(2):198--212, 2003.
[4]
Tiago Pessoa et al. Enhanced analog and RF IC sizing methodology using PCA and NSGA-II optimization kernel. In 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 660--665. IEEE.
[5]
Wenlong Lyu et al. Batch bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design. In International conference on machine learning, pages 3306--3314. PMLR, 2018.
[6]
Bo Liu et al. Gaspad: A general and efficient mm-wave integrated circuit synthesis method based on surrogate model assisted evolutionary algorithm. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 33(2):169--182, 2014.
[7]
Keertana Settaluri et al. Autockt: Deep reinforcement learning of analog circuit designs. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 490--495. IEEE, 2020.
[8]
Bobak Shahriari et al. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148--175, 2015.
[9]
Peter Frazier et al. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.
[10]
Hanrui Wang et al. Learning to design circuits. In NeurIPS Machine Learning for Systems Workshop, 2018.
[11]
Hanrui Wang et al. GCN-RL circuit designer: Transferable transistor sizing with graph neural networks and reinforcement learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1--6. IEEE, 2020.
[12]
N.S. Karthik Somayaji, Hanbin Hu, and Peng Li. Prioritized reinforcement learning for analog circuit optimization with design knowledge. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 1231--1236. IEEE.
[13]
Kai Yang et al. Trust-region method with deep reinforcement learning in analog design space exploration. In 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE.
[14]
Wei Shi et al. Robustanalog: Fast variation-aware analog circuit design via multi-task rl. In Proceedings of the 2022 ACM/IEEE Workshop on Machine Learning for CAD, pages 35--41, 2022.
[15]
Minjeong Choi et al. Reinforcement learning-based analog circuit optimizer using gm/id for sizing. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1--6. IEEE, 2023.
[16]
Jinxin Zhang et al. Automated design of complex analog circuits with multiagent based reinforcement learning. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1--6. IEEE, 2023.
[17]
Ahmet Budak et al. Dnn-Opt: An rl inspired optimization for analog circuit sizing using deep neural networks. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 1219--1224. IEEE, 2021.
[18]
Youngchang Choi et al. MA-Opt: Reinforcement learning-based analog circuit optimization using multi-actors. In 2023 Design, Automation Test in Europe Conference Exhibition (DATE), pages 1--5, 2023.
[19]
Kurtland Chua et al. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[20]
Michael Janner et al. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[21]
Wei Zhao et al. New generation of predictive technology model for sub-45 nm early design exploration. IEEE Transactions on electron Devices, 53(11):2816--2823, 2006.

Index Terms

  1. Using Probabilistic Model Rollouts to Boost the Sample Efficiency of Reinforcement Learning for Automated Analog Circuit Sizing
              Index terms have been assigned to the content through auto-classification.

              Recommendations

              Comments

              Information & Contributors

              Information

              Published In

              cover image ACM Conferences
              DAC '24: Proceedings of the 61st ACM/IEEE Design Automation Conference
              June 2024
              2159 pages
              ISBN:9798400706011
              DOI:10.1145/3649329
              Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

              Sponsors

              In-Cooperation

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              Published: 07 November 2024

              Check for updates

              Author Tags

              1. circuit design automation
              2. model based policy optimization
              3. twin delayed deep deterministic policy gradient
              4. analog circuit sizing

              Qualifiers

              • Research-article

              Funding Sources

              Conference

              DAC '24
              Sponsor:
              DAC '24: 61st ACM/IEEE Design Automation Conference
              June 23 - 27, 2024
              CA, San Francisco, USA

              Acceptance Rates

              Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

              Upcoming Conference

              DAC '25
              62nd ACM/IEEE Design Automation Conference
              June 22 - 26, 2025
              San Francisco , CA , USA

              Contributors

              Other Metrics

              Bibliometrics & Citations

              Bibliometrics

              Article Metrics

              • 0
                Total Citations
              • 115
                Total Downloads
              • Downloads (Last 12 months)115
              • Downloads (Last 6 weeks)24
              Reflects downloads up to 08 Feb 2025

              Other Metrics

              Citations

              View Options

              Login options

              View options

              PDF

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader

              Figures

              Tables

              Media

              Share

              Share

              Share this Publication link

              Share on social media