Abstract
It may be tempting to learn about scholars using ChatGPT. To validate ChatGPT for this task a small experiment was conducted based on the 50 most cited researchers at the author’s university. The results show that ChatGPT (GPT-3.5) only recognized an ad-hoc subset of the scholars with no obvious connection to the respective authors’ citation attributes or internet footprint. Moreover, details about the recognized scholars were often erroneous.
This is a preview of subscription content, log in via an institution to check access.
References
Aczel, B., & Wagenmakers, E. J. (2023). Transparency guidance for ChatGPT usage in scientific writing.
Carniel, T., Cazenille, L., Dalle, J. M., & Halloy, J. (2023). Topical Grouping of Thousands of Biomimetics Articles According to Their Goals, Results and Methods. In Conference on Biomimetic and Biohybrid Systems (pp. 257–272). Cham: Springer Nature Switzerland.
de Winter, J. (2023). Transforming Scientometric Analysis with GPT-4: A Study on Predicting Citations, Readership, and Social Media Interaction.
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M. ,Koohang, A.,Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A.,Al-Busaidi, A. S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., ... & Wright, R. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642.
Eika, E., & Sandnes, F. E. (2022). Starstruck by journal prestige and citation counts? On students’ bias and perceptions of trustworthiness according to clues in publication references. Scientometrics, 127(11), 6363–6390.
Farhat, F., Silva, E. S., Hassani, H., Madsen, D. Ø., Sohail, S. S., Himeur, Y., ... & Zafar, A. (2023). Analyzing the scholarly footprint of ChatGPT: Mapping the progress and identifying future trends.
Farhat, F., Sohail, S. S., & Madsen, D. Ø. (2023b). How trustworthy is ChatGPT? The case of bibliometric analyses. Cogent Engineering, 10(1), 2222988.
Flanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2023). Nonhuman “authors” and implications for the integrity of scientific publication and medical knowledge. JAMA, 329(8), 637–639.
Hosseini, M., & Horbach, S. P. (2023). Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research Integrity and Peer Review, 8(1), 4.
Khosravi, H., Shafie, M. R., Hajiabadi, M., Raihan, A. S., & Ahmed, I. (2023). Chatbots and ChatGPT: A bibliometric analysis and systematic review of publications in Web of Science and Scopus databases. http://arXiv.org/2304.05436
Kirtania, D. K. (2023). ChatGPT as a tool for Bibliometrics Analysis: Interview with ChatGPT. Available at SSRN 4391794.
Lotfigolian, M., Papanikolaou, C., Taghizadeh, S., & Sandnes, F. E. (2023). Human Experts’ Perceptions of Auto-Generated Summarization Quality. In Proceedings of the 16th International Conference on Pervasive Technologies Related to Assistive Environments (PETRA '23), July 05–07, 2023, Corfu, Greece (pp. 95–98). ACM, New York. https://doi.org/10.1145/3594806.3594828
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023). ChatGPT and a new academic reality: Artificial Intelligence-written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, 74(5), 570–581.
Orduña-Malea, E., & Cabezas-Clavijo, Á. (2023). ChatGPT and the potential growing of ghost bibliographic references. Scientometrics. https://doi.org/10.1007/s11192-023-04804-4
Pereira, V., Basilio, M. P., & Santos, C. H. T. (2023). pyBibX--A Python Library for Bibliometric and Scientometric Analysis Powered with Artificial Intelligence Tools. http://arXiv.org/2304.14516
Petiska, E. (2023). ChatGPT cites the most-cited articles and journals, relying solely on Google Scholar's citation counts. As a result, AI may amplify the Matthew Effect in environmental science. http://arXiv.org/2304.06794
Sandnes, F. E. (2020). A simple back-of-the-envelope test for self-citations using Google Scholar author profiles. Scientometrics, 124(2), 1685–1689.
Sandnes, F. E. (2021). Everyone onboard? Participation ratios as a metric for research activity assessments within young universities. Scientometrics, 126(7), 6105–6113.
Stokel-Walker, C. (2023). ChatGPT listed as author on research papers: Many scientists disapprove. Nature, 613(7945), 620–621.
Tang, L., Zhou, X., & Lu, M. (2023). A GPT-Based Approach for Scientometric Analysis: Exploring the Landscape of Artificial Intelligence Research. http://arXiv.org/2304.09487
Tomlinson, B., Torrance, A. W., & Black, R. W. (2023). ChatGPT and Works Scholarly: Best Practices and Legal Pitfalls in Writing with AI. http://arXiv.org/2305.03722
Acknowledgements
ChatGPT was used for formatting some of the references into APA style using the prompt “Format the following reference into APA: ….”.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author declares no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sandnes, F.E. Can we identify prominent scholars using ChatGPT?. Scientometrics 129, 713–718 (2024). https://doi.org/10.1007/s11192-023-04882-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04882-4