skip to main content
10.1145/3520304.3533961acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article
Public Access

Quantum neuron selection: finding high performing subnetworks with quantum algorithms

Published: 19 July 2022 Publication History

Abstract

Gradient descent methods have long been the de facto standard for training deep neural networks. Millions of training samples are fed into models with billions of parameters, which are slowly updated over hundreds of epochs. Recently, it's been shown that large, randomly initialized neural networks contain subnetworks that perform as well as fully trained models. This insight offers a promising avenue for training future neural networks by simply pruning weights from large, random models. However, this problem is combinatorically hard and classical algorithms are not efficient at finding the best subnetwork. In this paper, we explore how quantum algorithms could be formulated and applied to this neuron selection problem. We introduce several methods for local quantum neuron selection that reduce the entanglement complexity that large scale neuron selection would require, making this problem more tractable for current quantum hardware.

References

[1]
Scott Aaronson. 2018. Introduction to Quantum Information Science. https://www.scottaaronson.com/qclec.pdf
[2]
MD SAJID ANIS et al. 2021. Qiskit: An Open-source Framework for Quantum Computing.
[3]
B. Apolloni, C. Carvalho, and D. de Falco. 1989. Quantum stochastic optimization. Stochastic Processes and their Applications 33, 2 (1989), 233--244.
[4]
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. 2020. What is the State of Neural Network Pruning?. In Proceeding of the Machine Learning and Systems Conference.
[5]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners.
[6]
Stephen G. Brush. 1967. History of the Lenz-Ising Model. Reviews of Modern Physics 39, 4 (Oct. 1967), 883--893.
[7]
Hugh Collins and Kortney Easterly. 2021. IBM Unveils Breakthrough 127-Qubit Quantum Processor. https://newsroom.ibm.com/2021-11-16-IBM-Unveils-Breakthrough-127-Qubit-Quantum-Processor
[8]
Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. 2021. CoAtNet: Marrying Convolution and Attention for All Data Sizes. arXiv:2106.04803 [cs.CV]
[9]
Diego de Falco, B. Apolloni, and Nicolò Cesa-Bianchi. 1988. A numerical implementation of quantum annealing.
[10]
James Diffenderfer and Bhavya Kailkhura. 2021. Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network.
[11]
Gerald M. Edelman. 1987. Neural Darwinism : the theory of neuronal group selection. Basic Books. http://www.worldcat.org/isbn/9780465049349
[12]
Edward Farhi, Jeffrey Goldstone, and Sam Gutmann. 2014. A Quantum Approximate Optimization Algorithm.
[13]
Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv:1803.03635 [cs.LG]
[14]
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. 2019. Linear Mode Connectivity and the Lottery Ticket Hypothesis.
[15]
Adam Gaier and David Ha. 2019. Weight Agnostic Neural Networks. arXiv:1906.04358 [cs.LG]
[16]
Lov K. Grover. 1996. A Fast Quantum Mechanical Algorithm for Database Search. In Proceedings of the Twenty-Eighth Annual ACM Symposium on Theory of Computing (Philadelphia, Pennsylvania, USA) (STOC '96). Association for Computing Machinery, New York, NY, USA, 212--219.
[17]
Laszlo Gyongyosi and Sandor Imre. 2019. A Survey on quantum computing technology. Computer Science Review 31 (2019), 51--71.
[18]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network.
[19]
Herbert Jaeger and Harald Haas. 2004. Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication. Science 304, 5667 (2004), 78--80. arXiv:https://www.science.org/doi/pdf/10.1126/science.1091277
[20]
Yann LeCun, John Denker, and Sara Solla. 1989. Optimal Brain Damage. In Advances in Neural Information Processing Systems, D. Touretzky (Ed.), Vol. 2. Morgan-Kaufmann. https://proceedings.neurips.cc/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf
[21]
Eran Malach, Gilad Yehudai, Shai Shalev-Shwartz, and Ohad Shamir. 2020. Proving the Lottery Ticket Hypothesis: Pruning is All You Need.
[22]
Catherine McGeoch and Pau Farré. 2022. Advantage Processor Overview. https://www.dwavesys.com/media/3xvdipcn/14-1048aa_advantage_processor_overview.pdf
[23]
Laurent Orseau, Marcus Hutter, and Omar Rivasplata. 2020. Logarithmic Pruning is All You Need. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 2925--2934. https://proceedings.neurips.cc/paper/2020/file/1e9491470749d5b0e361ce4f0b24d037-Paper.pdf
[24]
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. 2020. Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient.
[25]
Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man-Hong Yung, Xiao-Qi Zhou, Peter J. Love, Alán Aspuru-Guzik, and Jeremy L. O'Brien. 2014. A variational eigenvalue solver on a photonic quantum processor. Nature Communications 5, 1 (jul 2014).
[26]
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. 2020. What's Hidden in a Randomly Weighted Neural Network?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[27]
Michele Sasdelli and Tat-Jun Chin. 2021. Quantum Annealing Formulation for Binary Neural Networks.
[28]
Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. 2017. Failures of Gradient-Based Deep Learning.
[29]
Darrell Whitley, Renato Tinós, and Francisco Chicano. 2015. Optimal Neuron Selection: NK Echo State Networks for Reinforcement Learning.
[30]
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. 2019. Discovering Neural Wirings.
[31]
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2021. Scaling Vision Transformers. arXiv:2106.04560 [cs.CV]
[32]
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. 2019. Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask.

Index Terms

  1. Quantum neuron selection: finding high performing subnetworks with quantum algorithms
      Index terms have been assigned to the content through auto-classification.

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion
      July 2022
      2395 pages
      ISBN:9781450392686
      DOI:10.1145/3520304
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 July 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. machine learning
      2. neural networks
      3. neuron selection
      4. nk landscapes
      5. quantum computing

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      GECCO '22
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 135
        Total Downloads
      • Downloads (Last 12 months)72
      • Downloads (Last 6 weeks)8
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media