short-paper

Deep Reinforcement Learning Methods for Discovering Novel Neuromorphic Devices

Authors:

Douglas Cale Crowder,

John Darby Smith,

Suma George CardwellAuthors Info & Claims

ICONS '23: Proceedings of the 2023 International Conference on Neuromorphic Systems

Article No.: 37, Pages 1 - 8

https://doi.org/10.1145/3589737.3605979

Published: 28 August 2023 Publication History

Abstract

Innovation in the field of neuromorphic computing is characterized by long periods of slow, steady growth that are punctuated by periods of rapid discovery. In order to accelerate discovery in the field of neuromorphic computing and disrupt the process of slow and steady growth, we are proposing a reinforcement learning architecture that is able to automatically construct simple circuits. Rather than providing the reinforcement learning agent with specifications for existing devices, we ask the reinforcement learning agent to provide specifications for novel devices that, if fabricated, can solve a specified problem. We show that, by slightly changing the problem statement, we can cause the reinforcement learning agent to produce different device specifications. Ultimately, we expect that by generating many possible solutions, the reinforcement learning agent will accelerate innovation by stimulating insight into potential solutions for problems.

References

[1]

Muhammad Abrar, Ushna Ajmal, Ziyad M Almohaimeed, Xiang Gui, Rizwan Akram, and Roha Masroor. 2021. Energy efficient UAV-enabled mobile edge computing for IoT devices: a review. IEEE Access (2021).

[2]

James Aimone, Prasanna Date, Gabriel Fonseca-Guerra, Kathleen Hamilton, Kyle Henke, Bill Kay, Garrett Kenyon, Shruti Kulkarni, Susan Mniszewski, and Maryam Parsa. 2022. A review of non-cognitive applications for neuromorphic computing. Neuromorphic Computing and Engineering (2022).

[3]

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. Advances in neural information processing systems 30 (2017).

Digital Library

[4]

Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph C. Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando G. S. L. Brandao, David A. Buell, Brian Burkett, Yu Chen, Zijun Chen, Ben Chiaro, Roberto Collins, William Courtney, Andrew Dunsworth, Edward Farhi, Brooks Foxen, Austin Fowler, Craig Gidney, Marissa Giustina, Rob Graff, Keith Guerin, Steve Habegger, Matthew P. Harrigan, Michael J. Hartmann, Alan Ho, Markus Hoffmann, Trent Huang, Travis S. Humble, Sergei V. Isakov, Evan Jeffrey, Zhang Jiang, Dvir Kafri, Kostyantyn Kechedzhi, Julian Kelly, Paul V. Klimov, Sergey Knysh, Alexander Korotkov, Fedor Kostritsa, David Landhuis, Mike Lindmark, Erik Lucero, Dmitry Lyakh, Salvatore Mandrà, Jarrod R. McClean, Matthew McEwen, Anthony Megrant, Xiao Mi, Kristel Michielsen, Masoud Mohseni, Josh Mutus, Ofer Naaman, Matthew Neeley, Charles Neill, Murphy Yuezhen Niu, Eric Ostby, Andre Petukhov, John C. Platt, Chris Quintana, Eleanor G. Rieffel, Pedram Roushan, Nicholas C. Rubin, Daniel Sank, Kevin J. Satzinger, Vadim Smelyanskiy, Kevin J. Sung, Matthew D. Trevithick, Amit Vainsencher, Benjamin Villalonga, Theodore White, Z. Jamie Yao, Ping Yeh, Adam Zalcman, Hartmut Neven, and John M. Martinis. 2019. Quantum supremacy using a programmable superconducting processor. Nature 574, 7779 (2019), 505--510.

[5]

Geoffrey W Burr, Robert M Shelby, Abu Sebastian, Sangbum Kim, Seyoung Kim, Severin Sidler, Kumar Virwani, Masatoshi Ishii, Pritish Narayanan, Alessandro Fumarola, et al. 2017. Neuromorphic computing using non-volatile memory. Advances in Physics: X 2, 1 (2017), 89--124.

[6]

Douglas C Crowder, J Darby Smith, and Suma G Cardwell. 2023. AI-Enhanced Codesign of Neuromorphic Circuit. In 2023 IEEE 65th International Midwest Symposium on Circuits and Systems (MWSCAS). IEEE.

[7]

Steve Furber. 2016. Large-scale neuromorphic computing systems. Journal of neural engineering 13, 5 (2016), 051001.

[8]

Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. 2020. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence 44, 1 (2020), 154--180.

[9]

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural networks 2, 5 (1989), 359--366.

Digital Library

[10]

Andrew Jolley, Saeed Afshar, Greg Cohen, Richard Lazarus Pahlavani, and Andrew Lambert. 2023. Neuromorphic Sensor Event-Rate Monitoring for Satellite Characterization. Journal of Spacecraft and Rockets (2023), 1--12.

[11]

Norman Jouppi, Cliff Young, Nishant Patil, and David Patterson. 2018. Motivation for and Evaluation of the First Tensor Processing Unit. IEEE Micro 38, 3 (2018), 10--19.

[12]

Olga Krestinskaya, Alex Pappachen James, and Leon Ong Chua. 2019. Neuromemristive circuits for edge computing: A review. IEEE transactions on neural networks and learning systems 31, 1 (2019), 4--23.

[13]

Suhas Kumar, R Stanley Williams, and Ziwen Wang. 2020. Third-order nanocircuit elements for neuromorphic engineering. Nature 585, 7826 (2020), 518--523.

[14]

Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. 2021. A graph placement methodology for fast chip design. Nature 594, 7862 (2021), 207--212.

[15]

Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, Vol. 99. 278--287.

Digital Library

[16]

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1--8.

[17]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

[18]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.

Digital Library

[19]

Chetan Singh Thakur, Jamal Lottier Molin, Gert Cauwenberghs, Giacomo Indiveri, Kundan Kumar, Ning Qiao, Johannes Schemmel, Runchun Wang, Elisabetta Chicca, Jennifer Olson Hasler, et al. 2018. Large-scale neuromorphic spiking array processors: A quest to mimic the brain. Frontiers in neuroscience 12 (2018), 891.

[20]

Craig Vineyard, Suma Cardwell, Frances Chance, Srideep Musuvathy, Fred Rothganger, William Severa, John Smith, Corinne Teeter, Felix Wang, and James Aimone. 2022. Neural Mini-Apps as a Tool for Neuromorphic Computing Insight. In Neuro-Inspired Computational Elements Conference. 40--49.

Digital Library

[21]

Craig Vineyard, William Severa, Matthew Kagie, Andrew Scholand, and Park Hays. 2019. A Resurgence in Neuromorphic Architectures Enabling Remote Sensing Computation. In 2019 IEEE Space Computing Conference (SCC). IEEE, 33--40.

[22]

M Mitchell Waldrop. 2016. The chips are down for Moore's law. Nature News 530, 7589 (2016), 144.

[23]

Angel Yanguas-Gil, Jaehoon Koo, Sandeep Madireddy, Prasanna Balaprakash, Jeffrey W Elam, and Anil U Mane. 2021. Neuromorphic architectures for edge computing under extreme environments. In 2021 IEEE Space Computing Conference (SCC). IEEE, 39--45.

Cited By

Cardwell SPatel KSchuman CDarby Smith JKwon JMaicke AArzate JIncorvia J(2024)Device Codesign using Reinforcement Learning2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558165(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558165

Index Terms

Deep Reinforcement Learning Methods for Discovering Novel Neuromorphic Devices
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Hardware
  1. Electronic design automation
  2. Emerging technologies
    1. Analysis and design of emerging devices and systems
      1. Emerging architectures
      2. Emerging tools and methodologies

Recommendations

Discrete-to-deep reinforcement learning methods
Abstract
Neural networks are effective function approximators, but hard to train in the reinforcement learning (RL) context mainly because samples are correlated. In complex problems, a neural RL approach is often able to learn a better solution than ...
Deep Reinforcement Learning: From Q-Learning to Deep Q-Learning
Neural Information Processing
Abstract
As the two hottest branches of machine learning, deep learning and reinforcement learning both play a vital role in the field of artificial intelligence. Combining deep learning with reinforcement learning, deep reinforcement learning is a method ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICONS '23: Proceedings of the 2023 International Conference on Neuromorphic Systems

August 2023

270 pages

ISBN:9798400701757

DOI:10.1145/3589737

Conference Chair:
Catherine Schuman,
Program Co-chairs:
Melika Payvand,
Maryam Parsa

Copyright © 2023 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGDA: ACM Special Interest Group on Design Automation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Conference

ICONS '23

Sponsor:

SIGDA

ICONS '23: 2023 International Conference on Neuromorphic Systems

August 1 - 3, 2023

NM, Santa Fe, USA

Acceptance Rates

Overall Acceptance Rate 13 of 22 submissions, 59%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
58
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)6

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cardwell SPatel KSchuman CDarby Smith JKwon JMaicke AArzate JIncorvia J(2024)Device Codesign using Reinforcement Learning2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558165(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558165

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten