Skip to main content
Springer Nature Link
Log in

Quantifying the progress of artificial intelligence subdomains using the patent citation network

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Even though Artificial Intelligence (AI) has been having a transformative effect on human life, there is currently no precise quantitative method for measuring and comparing the performance of different AI methods. Technology Improvement Rate (TIR) is a measure that describes a technology’s rate of performance improvement, and is represented in a generalization of Moore’s Law. Estimating TIR is important for R&D purposes to forecast which competing technologies have a higher chance of success in the future. The present contribution estimates the TIR for different subdomains of applied and industrial AI by quantifying each subdomain’s centrality in the global flow of technology, as modeled by the Patent Citation Network and shown in previous work. The estimated TIR enables us to quantify and compare the performance improvement of different AI methods. We also discuss the influencing factors behind slower or faster improvement rates. Our results highlight the importance of Rule-based Machine Learning (not to be confused with Rule-based Systems), Multi-task Learning, Meta-Learning, and Knowledge Representation in the future advancement of AI and particularly in Deep Learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Abadi, H. H. N., & Pecht, M. (2020). Artificial intelligence trends based on the patents granted by the united states patent and trademark office. IEEE Access, 8, 81633–81643.

    Article  Google Scholar 

  • Abood, A., & Feltenberger, D. (2018). Automated patent landscaping. Artifical Intelligence and Law, 26, 103–125.

    Article  Google Scholar 

  • AIME Planning Team. (2021). Artificial intelligence measurement and evaluation at the national institute of standards and technology

  • Alstott, J., Triulzi, G., & Yan, B. (2016). Mapping technology space by normalizing patent networks. Scientometrics. https://doi.org/10.1007/s11192-016-2107-y

    Article  Google Scholar 

  • Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012

    Article  Google Scholar 

  • Baruffaldi, S.H., Baruffaldi, S., Rao, N. (2020). Identifying and measuring developments in artificial intelligence

  • Basnet, S., & Magee, C. L. (2017). Artifact interactions retard technological improvement: An empirical study. PLoS ONE, 12, 1–17. https://doi.org/10.1371/journal.pone.0179596

    Article  Google Scholar 

  • Batagelj, V. (2003). Efficient algorithms for citation network analysis. arXiv Prepr. 1–27

  • Bengio, Y., Lecun, Y., & Hinton, G. (2021). Deep learning for AI. Communications of the ACM, 64, 58–65.

    Article  Google Scholar 

  • Benson, C. L., & Magee, C. L. (2015a). Quantitative determination of technological improvement from patent data. PLoS ONE, 10, 1–23. https://doi.org/10.1371/journal.pone.0121635

    Article  Google Scholar 

  • Benson, C. L., & Magee, C. L. (2015b). Technology structural implications from the extension of a patent search method. Scientometrics, 102, 1965–1985.

    Article  Google Scholar 

  • Crevier, D. (1993). AI: The tumultuous history of the search for artificial intelligence. Basic Books Inc.

    Google Scholar 

  • Darwiche, A. (2020). Three modern roles for logic in AI. In: Proceedings of the 39th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems. pp. 229–243

  • Davis, R., Shrobe, H., & Szolovits, P. (1993). What is a knowledge representation? AI Magazine, 14, 17.

    Google Scholar 

  • Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv Prepr. arXiv1810.04805

  • European Patent Office (2019) Technical character of an invention, https://www.epo.org/en/legal/case-law/2019/clr_i_d_9_1_1.html

  • Giczy, A. V., Pairolero, N. A., & Toole, A. A. (2022). Identifying artificial intelligence (AI) invention: A novel AI patent dataset. The Journal of Technology Transfer, 47, 476–505. https://doi.org/10.1007/s10961-021-09900-2

    Article  Google Scholar 

  • Goyal, A., Didolkar, A., Ke, N. R., Blundell, C., Beaudoin, P., Heess, N., Mozer, M. C., & Bengio, Y. (2021). Neural production systems. Advances in Neural Information Processing Systems., 34, 25673–25687.

    Google Scholar 

  • Grace, K. (2013). Algorithmic progress in six domains. Machine Intelligence Research Institute.

  • Huisman, M., van Rijn, J. N., & Plaat, A. (2021). A survey of deep meta-learning. Artificial Intelligence Review. https://doi.org/10.1007/s10462-021-10004-4

    Article  Google Scholar 

  • Hummon, N. P., & Dereian, P. (1989). Connectivity in a citation network: The development of DNA theory. Social Networks, 11, 39–63.

    Article  Google Scholar 

  • Hurwitz, J., Kaufman, M., & Bowles, A. (2012). The foundation of cognitive computing. Cognitive computing and big data analytics (pp. 1–20). Wiley.

    Chapter  Google Scholar 

  • Jiang, L., Chen, J., Bao, Y., & Zou, F. (2022). Exploring the patterns of international technology diffusion in AI from the perspective of patent citations. Scientometrics, 127, 5307–5323. https://doi.org/10.1007/s11192-021-04134-3

    Article  Google Scholar 

  • Khayyam, H., Jamali, A., Bab-Hadiashar, A., Esch, T., Ramakrishna, S., Jalili, M., & Naebe, M. (2020). A novel hybrid machine learning algorithm for limited and big data modeling with application in industry 4.0. IEEE Access, 8, 111381–111393. https://doi.org/10.1109/ACCESS.2020.2999898

    Article  Google Scholar 

  • Lanzi, P., & Stolzmann, W. (2000). Learning classifier systems: From foundations to applications. Springer.

    Book  Google Scholar 

  • Lee, S., Hwang, J., & Cho, E. (2022). Comparing technology convergence of artificial intelligence on the industrial sectors: Two-way approaches on network analysis and clustering analysis. Scientometrics, 127, 407–452. https://doi.org/10.1007/s11192-021-04170-z

    Article  Google Scholar 

  • Leiserson, C. E., Thompson, N. C., Emer, J. S., Kuszmaul, B. C., Lampson, B. W., Sanchez, D., & Schardl, T. B. (2020). There’s plenty of room at the Top: What will drive computer performance after Moore’s law? Science, 80, 368. https://doi.org/10.1126/science.aam9744

    Article  Google Scholar 

  • Li, C. OpenAI’s GPT-3 Language model: A technical overview, https://lambdalabs.com/blog/demystifying-gpt-3/

  • Liu, N., Shapira, P., & Yue, X. (2021a). Tracking developments in artificial intelligence research: constructing and applying a new search strategy. Scientometrics, 126, 3153–3192. https://doi.org/10.1007/s11192-021-03868-4

    Article  Google Scholar 

  • Liu, N., Shapira, P., Yue, X., & Guan, J. (2021b). Mapping technological innovation dynamics in artificial intelligence domains: Evidence from a global patent analysis. PLoS ONE. https://doi.org/10.1371/journal.pone.0262050

    Article  Google Scholar 

  • Liu, Z., Lin, Y., & Sun, M. (2020). Representation learning for natural language processing. Springer.

    Book  Google Scholar 

  • Lohn, A.J., Musser, M. (2022). AI and compute: How much longer can computing power drive artificial intelligence progress?

  • Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., Wu, J. (2019). The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv Prepr. arXiv1904.12584

  • Marcus, G. (2018) Deep learning: A critical appraisal. arXiv Prepr. arXiv1801.00631

  • Marcus, G. (2020). The next decade in AI: Four steps towards robust artificial intelligence. arXiv e-prints

  • Mattson, P., Reddi, V. J., Cheng, C., Coleman, C., Diamos, G., Kanter, D., Micikevicius, P., Patterson, D., Schmuelling, G., Tang, H., Wei, G.-Y., & Wu, C.-J. (2020). MLPerf: An industry standard benchmark suite for machine learning performance. IEEE Micro, 40, 8–16. https://doi.org/10.1109/MM.2020.2974843

    Article  Google Scholar 

  • Meyer, M. (2000). What is special about patent citations? Differences between scientific and patent citations. Scientometrics, 49, 93–123. https://doi.org/10.1023/A:1005613325648

    Article  Google Scholar 

  • Mikolov, T., Chen, K., Corrado, G., Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv Prepr. arXiv1301.3781

  • Natarajan, S., Kersting, K., Khot, T., & Shavlik, J. (2014). Statistical relational learning. SpringerBriefs Computer Science. https://doi.org/10.1007/978-3-319-13644-8_2

    Article  Google Scholar 

  • OpenAI. (2022). AI and Compute. Blog Open AI. 1–11

  • Pan, Y. (2016). Heading toward artificial intelligence 20. Engineering, 2, 409–413. https://doi.org/10.1016/J.ENG.2016.04.018

    Article  Google Scholar 

  • Pandey, S., Verma, M. K., & Shukla, R. (2021). A scientometric analysis of scientific productivity of artificial intelligence research in India. Journal of Scientometric Research, 10, 245–250. https://doi.org/10.5530/JSCIRES.10.2.38

    Article  Google Scholar 

  • Patent and trademark office. (2019). 2019 revised patent subject matter eligibility guidance, https://www.federalregister.gov/documents/2019/01/07/2018-28282/2019-revised-patent-subject-matter-eligibility-guidance

  • Patterson, D., Gonzalez, J., Hölzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., & Dean, J. (2022). The carbon footprint of machine learning training will plateau, then shrink. Computer, 55, 18–28. https://doi.org/10.1109/MC.2022.3148714

    Article  Google Scholar 

  • Rejmaniak, R. (2021). Bias in artificial intelligence systems. Białostockie Studia Prawnicze, 3, 25–42.

    Article  Google Scholar 

  • Rudin, C., & Carlson, D. (2019). The secrets of machine learning: ten things you wish you had known earlier to be more effective at data analysis. Operations research & management science in the age of analytics management science in the age of analytics (pp. 44–72). INFORMS.

    Chapter  Google Scholar 

  • Sarker, M.K., Zhou, L., Eberhart, A., Hitzler, P. (2021). Neuro-symbolic artificial intelligence: current trends. arXiv e-prints

  • Sharifzadeh, M., Triulzi, G., & Magee, C. L. (2019). Quantification of technological progress in greenhouse gas (GHG) capture and mitigation using patent data. Energy & Environmental Science, 12, 2789–2805. https://doi.org/10.1039/c9ee01526d

    Article  Google Scholar 

  • Singh, A., Triulzi, G., & Magee, C. L. (2021). Technological improvement rate predictions for all technologies: Use of patent data and an extended domain description. Research Policy., 50, 104294. https://doi.org/10.1016/j.respol.2021.104294

    Article  Google Scholar 

  • Strubell, E., Ganesh, A., McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. ACL 2019—Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 3645–3650

  • Taheri, S., & Aliakbary, S. (2022). Research trend prediction in computer science publications: A deep neural network approach. Scientometrics, 127, 849–869. https://doi.org/10.1007/s11192-021-04240-2

    Article  Google Scholar 

  • Thompson, N.C., Greenewald, K., Lee, K., Manso, G.F. (2020) The computational limits of deep learning. arXiv e-prints

  • Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2021). Deep learning’s diminishing returns: the cost of improvement is becoming unsustainable. IEEE Spectrum, 58, 50–55. https://doi.org/10.1109/MSPEC.2021.9563954

    Article  Google Scholar 

  • Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C.C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P.S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv e-prints

  • Triulzi, G., Alstott, J., & Magee, C. L. (2020). Estimating technology performance improvement rates by mining patent data. Technological Forecasting and Social Change. https://doi.org/10.1016/j.techfore.2020.120100

    Article  Google Scholar 

  • Triulzi, G., & Magee, C. L. (2020). Functional performance improvement data and patent sets for 30 technology domains with measurements of patent centrality and estimations of the improvement rate. Data in Brief, 32, 106257. https://doi.org/10.1016/j.dib.2020.106257

    Article  Google Scholar 

  • Tseng, C.-Y., & Ting, P.-H. (2013). Patent analysis for technology development of artificial intelligence: A country-level comparative study. Innovation., 15, 463–475. https://doi.org/10.5172/impp.2013.15.4.463

    Article  Google Scholar 

  • van Engelen, J. E., & Hoos, H. H. (2020). A survey on semi-supervised learning. Machine Learning, 109, 373–440. https://doi.org/10.1007/s10994-019-05855-6

    Article  MathSciNet  Google Scholar 

  • Vanschoren, J. (2018). Meta-learning: a survey. 1–29

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł, & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5999–6009.

    Google Scholar 

  • Verendel, V. (2023). Tracking artificial intelligence in climate inventions with patent data. Nature Clinical Practice Endocrinology & Metabolism, 13, 40–47. https://doi.org/10.1038/s41558-022-01536-w

    Article  Google Scholar 

  • (2019). WIPO: WIPO Technology Trends 2019: Artificial Intelligence. World Intellectual Property Organization

  • Yu, C., & Yao, W. (2017). Robust linear regression: A review and comparison. Communications in Statistics - Simulation and Computation, 46, 6261–6282. https://doi.org/10.1080/03610918.2016.1202271

    Article  MathSciNet  Google Scholar 

  • Zhang, Y., & Yang, Q. (2021). A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2021.3070203

    Article  Google Scholar 

  • Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z. (2023). A survey of large language models. arXiv Prepr. arXiv2303.18223

Download references

Acknowledgements

We would like to thank Patents View and WIPO Patentscope staff for answering our questions. We would also like to thank Sara Mashhoon for reading a draft of this paper and Giorgio Triulzi for answering a question of ours. R.R. would like to thank his wife for her support during the completion of this research.

Author information

Authors and Affiliations

  1. Sharif Energy, Water and Environment Institute (SEWEI), Sharif University of Technology, Tehran, Iran

    Reza Rezazadegan & Mahdi Sharifzadeh

  2. Department of Mathematics, College of Sciences, Shiraz University, Shiraz, Iran

    Reza Rezazadegan

  3. Institute for Data, Systems, and Society, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA

    Christopher L. Magee

Authors
  1. Reza Rezazadegan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mahdi Sharifzadeh
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Christopher L. Magee
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Mahdi Sharifzadeh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 1242 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rezazadegan, R., Sharifzadeh, M. & Magee, C.L. Quantifying the progress of artificial intelligence subdomains using the patent citation network. Scientometrics 129, 2559–2581 (2024). https://doi.org/10.1007/s11192-024-04996-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-024-04996-3

Keywords

Mathematics Subject Classification

JEL Classification