skip to main content
10.1145/3658835.3658839acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaieeConference Proceedingsconference-collections
research-article

The Luckiest Network Gives the Average Error on Disjoint Tests: Experiments

Published: 12 June 2024 Publication History

Abstract

This is an experimental paper associated with the theoretical paper Weng [34] addressing the issue of “Deep Learning” misconduct in particular and Post-Selection in general because Post-Selection has contaminated beyond Deep Learning. Regardless of learning modes, almost all machine learning methods (except for a few methods that train a sole system) are rooted in the same misconduct— cheating and hiding—(1) cheating in the absence of a test and (2) hiding bad-looking data. The remaining open question is what is the expected error if the absence of a test in Misconduct (1) is corrected by conducting a test. Weng [34] has theoretically and mathematically proven that the expected error of the luckiest network on the validation set is the average of all trained networks including those bad-looking networks that were hidden by Misconduct (2). We conducted experiments in realistic synthetic environments, where a robot navigates using its camera. The virtual robot is controlled by a CNN-LSTM system that consists of a Convolution Neural Network (CNN) and a Long Short-Term Memory (LSTM). Compared with Developmental Networks (DN), the CNN-LSTM performed considerably worse than DN in new tests. This is true even when the luckiest on the validation set is compared with the sole DN. The luckiest CNN-LSTM performed indeed only like the average of all trained LSTM networks, including good-luck ones and bad-luck ones on the validation set. This first-ever experimental paper has experimentally confirmed the theoretical and mathematically proven results in [34]. Namely, for a realistic AI problem, the luck on a random sample (a validation set) does not transfer to another random sample (a disjoint test).

References

[1]
A. Aglinskas, J. K. Hartshorne, and S. Anzellotti. 2022. Contrastive machine learning reveals the structure of neuroanatomical variation within autism. Science 376, 6597 (June 2022), 1070–1074.
[2]
M. G. Bellemare, S. Candido, Z. Wang, 2020. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588, 7836 (2020), 77–82.
[3]
J. Bures and I. Larrosa. 2023. Organic reaction mechanism classification using machine learning. Nature 613, 7945 (Jan. 26 2023), 689–695.
[4]
E. Callaway. 2022. What’s Next for the AI Protein-Folding Revolution. Nature 604, 7905 (2022), 234–238.
[5]
K. Course and P. B. Nair. 2023. State estimation of a physical system with unknown governing equations. Nature 622, 7982 (2023), 261–267.
[6]
A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. 2021. First return, then explore. Nature 590, 7847 (Feb. 25 2021), 580–586.
[7]
M. Galvani. 2019. History and future of driver assistance. IEEE Instrumentation & Measurement Magazine 22, 1 (2019), 11–16.
[8]
Q. Gao, G. A. Ascoli, and L. Zhao. 2021. BEAN: Interpretable and efficient learning with biologically-enhanced artificial neuronal assembly regularization. Front. Neurorobot 15 (June 1 2021), 1–13. https://doi.org/10.3389/fnbot.2021.567482.
[9]
A. Graves, G. Wayne, M. Reynolds, D. Hassabis, 2016. Hybrid computing using a neural network with dynamic external memory. Nature 538 (2016), 471–476.
[10]
H. Y. Huang, R. Kueng, J. Preskill, 2022. Provably efficient machine learning for quantum many-body problems. Science 377, 6613 (Sept. 23 2022), 1397.
[11]
I. R. Humphreys, J. Pei, M. Baek, D. Baker, 2021. Computed structures of core eukaryotic protein complexes. Science 374, 6573 (2021), 1340.
[12]
M. I. Jordan and T. M. Mitchell. 2015. Machine learning: Trends, perspectives, and prospects. Science 349 (July 17 2015), 255–260.
[13]
A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.
[14]
Y. LeCun, L. Bengio, and G. Hinton. 2015. Deep Learning. Nature 521 (2015), 436–444.
[15]
H. Lu, D. J. Diaz, H. S. Alper, 2022. Machine learning-aided engineering of hydrolases for PET depolymerization. Nature 604, 7907 (2022), 662–667.
[16]
D. J. Mankowitz, D. J. Michi, D. Silver, 2023. Faster sorting algorithms discovered using deep reinforcement learning. Nature 618 (2023), 257–263.
[17]
S. M. McKinney, M. Sieniek, V. Godbole, S. Shetty, 2020. Int’l evaluation of an AI system for breast cancer screening. Nature 577 (2020), 89–94.
[18]
D. S. Modha, F. Akopyan, T. Ueda, 2023. Neural inference at the frontier of energy, space, and time. Science 382, 6668 (2023), 329–335.
[19]
S. M. Mousavi and G. C. Beroza. 2022. Deep-learning seismology. Science 377, 6607 (2022), 508–513.
[20]
S. Pai, Z. Sun, D. Miller, 2023. Experimentally realized in situ backpropagation for deep learning in photonic neural networks. Science 380 (2023), 398–404.
[21]
N. I. Rinehart, R. K. Saunthwal, S. E. Denmark, 2023. A machine-learning tool to predict substrate-adaptive conditions for Pd-catalyzed C-N couplings. Science 381, 6661 (2023), 965–972.
[22]
J. Schrittwieser, I. Antonoglou, D. Silver, 2020. Mastering Atari, Go, chess and shogi by planning with a learned model. Science 588, 7839 (2020), 604–609.
[23]
A. W. Senior, R. Evans, D. Hassabis, 2020. Improved protein structure prediction using potentials from deep learning. Nature 577 (2020), 706–710.
[24]
D. Silver, T. Hubert, D. Hassabis, 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140–1144.
[25]
D. Silver, J. Schrittwieser, D. Hassabis, 2017. Mastering the game of Go without human knowledge. Nature 550 (2017), 354–359.
[26]
N. Slonim, Y. Bilu, C. Alzate, R. Aharonov, 2021. An autonomous debating system. Nature 591, 7850 (March 18 2021), 379–384.
[27]
M. Tracy, O. Snitser, I. Yelin, R. Kishony, 2022. Minimizing treatment-induced emergence of antibiotic resistance in bacterial infections. Science 375, 6583 (Feb. 2022), 889–894.
[28]
J. Wang, S. Lisanza, D. Juergens, D. Baker, 2022. Scaffolding protein functional sites using deep learning. Science 377, 6604 (2022), 387–394.
[29]
S. Warnat-Herresthal, H. Schultze, K. L. Shastry, J. L. Schultze, 2021. Swarm Learning for decentralized and confidential clinical machine learning. Nature 594, 7862 (2021), 265–270.
[30]
J. Weng. 2015. Brain as an Emergent Finite Automaton: A Theory and Three Theorems. Int’l Journal of Intelligence Science 5, 2 (2015), 112–131.
[31]
J. Weng. 2021. On Post Selections Using Test Sets (PSUTS) in AI. In Proc. Int’l Joint Conference on Neural Networks. IEEE Press, Shenzhen, China, 1–8.
[32]
J. Weng. 2022. 20 million-dollar problems for any brain models and a holistic solution: Conscious learning. In Proc. Int’l Joint Conference on Neural Networks. IEEE Press, Padua, Italy, 1–9. http://www.cse.msu.edu/ weng/research/20M-IJCNN2022rvsd-cite.pdf.
[33]
J. Weng. 2022. On ‘Deep Learning’ Misconduct. In Proc. 2022 3rd International Symposium on Automation, Information and Computing (ISAIC 2022). SciTePress, Beijing, China, 1–8. arXiv:2211.16350.
[34]
J. Weng. 2023. Transparentizing Post-Selection in Deep Learning:Post-Selection is Worse than Average. In Proc. Workshop on Transparentizing Deep Learning, 2024 The 5th International Conference on Artificial Intelligence in Electronics Engineering (AIEE 2024). ACM Press, Bangkok, Thailand, 1–10. Under peer review.
[35]
J. Weng. 2023. Why Deep Learning’s Performance Data Are Misleading. In 2023 4th Int’l Conf. on Artificial Intelligence in Electronics Engineering. ACM Press, Haikou, China, 1–10. arXiv:2208.11228.
[36]
J. Weng, N. Ahuja, and T. S. Huang. 1997. Learning recognition and segmentation using the Cresceptron. Int’l Journal of Computer Vision 25, 2 (Nov. 1997), 109–143.
[37]
J. Weng and M. Luciw. 2009. Dually Optimal Neuronal Layers: Lobe Component Analysis. IEEE Trans. Autonomous Mental Development 1, 1 (2009), 68–85.
[38]
F. R. Willett, D. T. Avansino, K. V. Shenoy, 2021. High-performance brain-to-text communication via handwriting. Nature 593, 7858 (2021), 249–254.
[39]
F. R. Willett, E. M. Kunz, J. M. Henderson, 2023. A high-performance speech neuroprosthesis. Nature 620, 7976 (2023), 1031–1036.
[40]
X. Wu and J. Weng. 2021. On Machine Thinking. In Proc. Int’l Joint Conf. Neural Networks. IEEE Press, Shenzhen, China, 1–8.
[41]
Z. Zheng X. Wu and J. Weng. 2022. Developmental Network-2: the Autonomous Generation of Optimal Internal-Representation Hierarchy. IEEE Transactions on Neural Networks and Learning Systems 33, 11 (2022), 6867–6880.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
AIEE '24: Proceedings of the 2024 5th International Conference on Artificial Intelligence in Electronics Engineering
January 2024
89 pages
ISBN:9798400716850
DOI:10.1145/3658835
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deep Learning
  2. Machine Learning
  3. Misconduct
  4. Neural Networks
  5. Post-Selection

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • The Fundamental Research Funds for Central Universities

Conference

AIEE 2024

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 11
    Total Downloads
  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media