Skip to main content
SpringerLink
Log in

Toward the third generation artificial intelligence

  • Review
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

There have been two competing paradigms in artificial intelligence (AI) development ever since its birth in 1956, i.e., symbolism and connectionism (or sub-symbolism). While symbolism dominated AI research by the end of 1980s, connectionism gained momentum in the 1990s and is gradually displacing symbolism. This paper considers symbolism as the first generation of AI and connectionism as the second generation. However, each of these two paradigms simulates the human mind from only one perspective. AI cannot achieve true human behaviors by relying on only one paradigm. In order to develop novel AI technologies that are safe, reliable, and extensible, it is necessary to establish a new explainable and robust AI theory. To this end, this paper looks toward developing a third generation artificial intelligence by combining the current paradigms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Simon H A. Models of Man. New York: Wiley & Sons, 1957

    Google Scholar 

  2. Newell A, Simon H A. Computer science as empirical inquiry: symbols and search. Commun ACM, 1976, 19: 113–126

    Article  MathSciNet  Google Scholar 

  3. Newell A. Physical Symbol Systems. Cogn Sci, 1980, 4: 135–183

    Google Scholar 

  4. Fodor J A. Methodological solipsism considered as a research strategy in cognitive psychology. Behav Brain Sci, 1980, 3: 63–73

    Article  Google Scholar 

  5. McCarthy J, Minsky M L, Rochester N, et al. A proposal for the Dartmouth summer research project on artificial intelligence. 1955, 27: 4

  6. Lindsay, Robert K, Bruce G. Applications of Artificial Intelligence for Organic Chemistry: The Dendral Project. New York: McGraw-Hill Book Company, 1980

    Google Scholar 

  7. Buchanan B G, Shortliffe E H. Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Boston: Addison Wesley, 1984

    Google Scholar 

  8. Muggleton S, de Raedt L. Inductive logic programming: theory and methods. J Logic Programm, 1994, 19–20: 629–679

    Article  MathSciNet  Google Scholar 

  9. Riguzzi F, Bellodi E, Zese R. A history of probabilistic inductive logic programming. Front Robot AI, 2014, 1: 6

    Article  Google Scholar 

  10. Yang Q, Zhang Y, Dai W Y, et al. Transfer Learning. Cambridge: Cambridge University Press, 2020

    Book  Google Scholar 

  11. Ehrlinger L, Wolfram W. Towards a definition of knowledge graphs. In: Joint Proceedings of the Posters and Demos Track of 12th International Conference on Semantic Systems (SEMANTiCS2016) and 1st International Workshop on Semantic Change & Evolving Semantics (SuCCESS16), Leipzig, 2016

  12. Russel S J, Norvig P. Artificial Intelligence: A Modern Approach. 2nd ed. New York: Pearson Education Inc., 2003

    Google Scholar 

  13. Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev, 1958, 65: 6

    Article  Google Scholar 

  14. Rosenblatt F. Principles of Neurodynamics. Arch Gen Psychiatry, 1962 7: 218–219

    Article  Google Scholar 

  15. McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys, 1943, 5: 115–133

    Article  MathSciNet  Google Scholar 

  16. Hebb D O. The Organization of Behavior: A Neuropsychological Theory. London: Psychology Press, 1949

    Google Scholar 

  17. Minsky M, Papert S A. Perceptrons: An Introduction to Computational Geometry. Cambridge: MIT Press, 1969

    Google Scholar 

  18. Cauchy A. Methode generale pour la resolution des systemes d’equations simultanees. Comp Rend Acad Sci, Paris, 1847, 25: 536–538

    Google Scholar 

  19. Nesterov Y E. A method for solving the convex programming problem with convergence rate O(1/k2). Soviet Math Doklady, 1983, 27: 372–376

    Google Scholar 

  20. Linnainmaa S. The representation of the cumulative rounding error of an algorithm as a taylor expansion of the local rounding errors (in Finnish). Dissertation for Master’s Degree. Helsinki: University of Helsinki, 1970

    Google Scholar 

  21. Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors. Nature, 1986, 323: 533–536

    Article  Google Scholar 

  22. Janocha K, Czarnecki W M. On loss functions for deep neural networks in classification. Schedae Inform, 2016, 25: 49–59

    Google Scholar 

  23. Wan L, Zeiler M, Zhang A X, et al. Regularization of neural networks using DropConnect. In: Proceedings of the 30th International Conference on Machine Learning (ICML), Atlanta, 2013

  24. Fukushima K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybernetics, 1980, 36: 193–202

    Article  Google Scholar 

  25. Derevyanko G, Grudinin S, Bengio Y, et al. Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 2018, 34: 4046–4053

    Article  Google Scholar 

  26. Lipton J C, Berkowitz J. A critical review of recurrent neural networks for sequence learning. 2015. ArXiv:1506.00019v4

  27. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation, 1997, 9: 1735–1780

    Article  Google Scholar 

  28. Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets. Neural Comput, 2006, 18: 1527–1554

    Article  MathSciNet  Google Scholar 

  29. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521: 436–444

    Article  Google Scholar 

  30. Coulom R. Efficient selectivity and backup operators in Monte-Carlo tree search. In: Proceedings of International Conference on Computers and Games, Berlin, 2006

  31. Sutton R S, Barto A G. Reinforcement Learning: An Introduction. Cambridge: MIT Press, 2015

    Google Scholar 

  32. Dong Y P, Liao F Z, Pang T Y, et al. Boosting adversarial attacks with momentum. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018

  33. Dong Y, Su H, Zhu J, et al. Towards interpretable deep neural networks by leveraging adversarial examples. In: Proceedings of IJCAI Workshop on AISC, Sydney, 2019

  34. Ferrucci D, Brown E, Chu-Carroll J, et al. Building Watson: an overview of the DeepQA project. In: Proceedings of Association for the Advancement of Artificial Intelligence (AAAI), Atlanta, 2010

  35. Harnad S. The symbol grounding problem. Phys D-Nonlin Phenom, 1990, 42: 335–346

    Article  Google Scholar 

  36. Chen X, Duan Y, Houthooft R, et al. InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Barcelona, 2016. 2172–2180

  37. Liu Y, Wei F Y, Shao J, et al. Exploring disentangled feature representation beyond face identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Salt Lake City, 2018. 2080–2089

  38. Higgins I, Matthey L, Pal A, et al. Beta-VAE: learning basic visual concepts with a constrained variational framework. In: Proceedings of the 32nd International Conference on Logic Programming (ICLP), New York City, 2016

  39. Siddharth N, Paige B, Desmaison A, et al. Inducing interpretable representations with variational autoencoders. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Barcelona, 2016

  40. Li C, Xu K, Zhu J, et al. Triple generative adversarial nets. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Long Beach, 2017

  41. Hu X L, Zhang J W, Li J M, et al. Sparsity regularized HMAX for visual recognition. Plos One, 2014, 9: e81813

    Article  Google Scholar 

  42. Silver D, Singh S, Precup D, et al. Reward is enough. Artif Intell, 2021, 299: 103535

    Article  MathSciNet  Google Scholar 

  43. Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518: 529–533

    Article  Google Scholar 

  44. Vinyals O, Babuschkin I, Czarnecki W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 2019, 575: 350–354

    Article  Google Scholar 

  45. Silver D, Huang A, Maddison C J, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529: 484–489

    Article  Google Scholar 

  46. Silver D, Schrittwieser J, Simonyan K, et al. Mastering the game of Go without human knowledge. Nature, 2017, 550: 354–359

    Article  Google Scholar 

  47. Duan Y, Chen X, Houthooft R, et al. Benchmarking deep reinforcement learning for continuous control. In: Proceedings of International Conference on Machine Learning (ICML), New York, 2016

  48. Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. 2017. ArXiv:1707.06347

  49. Sutton R S, Barto A G. Reinforcement Learning: An Introduction. 2nd ed. Cambridge: MIT Press, 2015

    Google Scholar 

  50. François-Lavet V, Henderson P, Islam R, et al. An introduction to deep reinforcement learning. FNT Machine Learn, 2018, 11: 219–354

    Article  Google Scholar 

  51. Huang S Y, Su H, Zhu J, et al. SVQN: sequential variational soft Q-learning networks. In: Proceedings of International Conference on Learning Representations (ICLR), 2020

  52. Huang S Y, Su H, Zhu J, et al. Combo-action: training agent for FPS game with auxiliary tasks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), Honolulu, 2019

  53. Song S H, Weng J Y, Su H, et al. Playing FPS games with environment-aware hierarchical reinforcement learning. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), Macau, 2019

  54. Zhou Y C, Li J L, Zhu J. Posterior sampling for multi-agent reinforcement learning: solving extensive games with imperfect information. In: Proceedings of International Conference on Learning Representations (ICLR), Addis Ababa, 2020

  55. Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space. In: Proceedings of International Conference on Learning Representations (ICLR), Scottsdale, 2013

  56. Pennington J, Socher R, Manning C D. Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, 2014

  57. Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Lake Tahoe, 2013

  58. Kusner M J, Sun Y, Kolkin N I, et al. From word embeddings to document distances. In: Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, 2015

  59. Zhang J, Liu Y, Luan H B, et al. Prior knowledge integration for neural machine translation using posterior regularization. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017

  60. Ding Y, Liu Y, Luan H B, et al, Visualizing and understanding neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017

  61. Zhang B. Artificial intelligence in the post-deep learning era (in Chinese). CAAI Trans Intell Technol, 2017, 7: 3–5

    Google Scholar 

  62. Liu M, Shi J, Li Z, et al. Towards better analysis of deep convolutional neural networks. In: Proceedings of IEEE Conference on Visual Analytics Science and Technology, 2016

  63. Wang Y L, Su H, Hu X L. Interpret neural networks by identifying critical data routing paths. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018

  64. Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Montreal, 2014

  65. Dong Y P, Su H, Zhu J, et al. Improving interpretability of deep neural networks with semantic information. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 2017

  66. Dong Y P, Fu Q-A, Yang X, et al. Benchmarking adversarial robustness on image classification. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 2020

  67. Dziugaite G K, Ghahramani Z, Roy D M. A study of the effect of jpg compression on adversarial images. 2016. ArXiv:1608.00853

  68. Samangouei P, Kabkab M, Chellappa R. Defense-GAN: protecting classifiers against adversarial attacks using generative models. In: Proceedings of International Conference on Learning Representations (ICLR), 2018

  69. Liao F Z, Liang M, Dong Y P, et al. Defense against adversarial attacks using high-level representation guided denoiser. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018

  70. Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks. In: Proceedings of International Conference on Learning Representations (ICLR), 2018

  71. Tramer F, Carlini N, Brendel W, et al. On adaptive attacks to adversarial example defenses. 2020. ArXiv:2002.08347

  72. Dong Y P, Deng Z J, Pang T Y, et al. Adversarial distributional training for robust deep learning. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020

  73. Kurakin A, Goodfellow I, Bengio S, et al. Adversarial attacks and defences competition. 2018. ArXiv:1804.00097

  74. Liu X Q, Cheng M H, Zhang H, et al. Towards robust neural networks via random selfensemble. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018

  75. Pang T Y, Xu K, Du C, et al. Improving adversarial robustness via promoting ensemble diversity. In: Proceedings of International Conference on Machine Learning (ICML), Long Beach, 2019

  76. Castelvecchi D. Can we open the black box of AI? Nature, 2016, 538: 20–23

    Article  Google Scholar 

  77. Ghahramani Z. Probabilistic machine learning and artificial intelligence. Nature, 2015, 521: 452–459

    Article  Google Scholar 

  78. Zhu J, Chen J, Hu W, et al. Big learning with Bayesian methods. Natl Sci Rev, 2017, 4: 627–651

    Article  Google Scholar 

  79. Zhu J, Chen N, Xing E P. Bayesian inference with posterior regularization and applications to infinite latent SVMs. J Mach Learn Res, 2014, 15: 1799–1847

    MathSciNet  Google Scholar 

  80. Williams P M. Bayesian conditionalisation and the principle of minimum information. Br J Philosophy Sci, 1980, 31: 131–144

    Article  MathSciNet  Google Scholar 

  81. Mei S, Zhu J, Zhu X. Robust RegBayes: selectively incorporating first-order logic domain knowledge into Bayesian models. In: Proceedings of International Conference on Machine Learning (ICML), Beijing, 2014

  82. Zhu J, Ahmed A, Xing E P. MedLDA: maximum margin supervised topic models. J Mach Learn Res, 2012, 13: 2237–2278

    MathSciNet  Google Scholar 

  83. MacKay D J C. Bayesian methods for adaptive models. Dissertation for Ph.D. Degree. Pasadena: California Institute of Technology, 1992

    Google Scholar 

  84. Neal R M. Bayesian learning for neural networks. Dissertation for Ph.D. Degree. Toronto: University of Toronto, 1995

    Google Scholar 

  85. Shi J, Sun S, Zhu J. A spectral approach to gradient estimation for implicit distributions. In: Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, 2018

  86. Zhou Y, Shi J, Zhu J. Nonparametric score estimators. In: Proceedings of International Conference on Machine Learning (ICML), Vienna, 2020

  87. Wang Z, Ren T, Zhu J, et al. Function space particle optimization for Bayesian neural networks. In: Proceedings of International Conference on Learning Representations (ICLR), New Orleans, 2019

  88. Xu K, Li C, Zhu J, et al. Understanding and stabilizing GANs’ training dynamics using control theory. In: Proceedings of International Conference on Machine Learning (ICML), Vienna, 2020

  89. Chen J, Lu C, Chenli B, et al. VFlow: more expressive generative flows with variational data augmentation. In: Proceedings of International Conference on Machine Learning (ICML), Vienna, 2020

  90. Shi J, Chen J, Zhu J, et al. ZhuSuan: a library for Bayesian deep learning. 2017. ArXiv:1709.05870

  91. Li J Z, Su H, Zhu J, et al. Essay-anchor attentive multi-modal bilinear pooling for textbook question answering. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME), San Diego, 2018

  92. Ren M, Kiros R, Zemel R S. Exploring models and data for image question answering. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2015

  93. Zhu Y, Groth O, Bernstein M S, et al. Visual7W: grounded question answering in images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 61620106010).

Author information

Authors and Affiliations

  1. Institute for Artificial Intelligence, Tsinghua University, Beijing, 100084, China

    Bo Zhang, Jun Zhu & Hang Su

Authors
  1. Bo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hang Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, B., Zhu, J. & Su, H. Toward the third generation artificial intelligence. Sci. China Inf. Sci. 66, 121101 (2023). https://doi.org/10.1007/s11432-021-3449-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-021-3449-x

Keywords

Search

Navigation