Skip to main content
SpringerLink
Log in

iSEER: an intelligent automatic computer system for scientific evaluation of researchers

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Funding is one of the crucial drivers of scientific activities. The increasing number of researchers and the limited financial resources have caused a tight competition among scientists to secure research funding. On the other side, it is now even harder for funding allocation organizations to select the most proper researchers. Number of publications and citation counts based indicators are the most common methods in the literature for analyzing the performance of researchers. However, the mentioned indicators are highly correlated with the career age and reputation of the researchers, since they accumulate over time. This makes it almost impossible to evaluate the performance of a researcher based on quantity and impact of his/her articles at the time of the publication. This article proposes an intelligent machine learning framework for scientific evaluation of researchers (iSEER). iSEER may help decision makers to better allocate the available funding to the distinguished scientists through providing fair comparative results, regardless of the career age of the researchers. Our results show that iSEER performs well in predicting the performance of the researchers with high accuracy, as well as classifying them based on collaboration patterns, research performance, and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. The NSERC database originally contains both scholarships and grants.

  2. Scopus is a commercial database of scientific articles that has been launched by Elsevier in 2004. It is now one of the main competitors of Thomson Reuter‘s Web of Science.

  3. We developed a new data extraction methodology which involves a combined use of Google Scholar and Scopus. The main idea is to use the full text search available in Google Scholar and then to search the results in the Scopus database and collect the target articles.

  4. Social network analysis software, for more information see: http://vlado.fmf.uni-lj.si/pub/networks/pajek/.

  5. Machine learning technique for topic modeling, first introduced by Blei et al. (2003).

References

  • Abbasi, A., Altmann, J., & Hossain, L. (2011). Identifying the effects of co-authorship networks on the performance of scholars: A correlation and regression analysis of performance measures and social network analysis measures. Journal of Informetrics, 5(4), 594–607.

    Article  Google Scholar 

  • Allen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. (2009). Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS One, 4(6), e5910.

    Article  Google Scholar 

  • Beaudry, C., & Allaoui, S. (2012). Impact of public and private research funding on scientific production: The case of nanotechnology. Research Policy, 41(9), 1589–1606.

    Article  Google Scholar 

  • Bell, J. G., & Seater, J. J. (1978). Publishing performance: Departmental and individual. Economic Inquiry, 16(4), 599–615.

    Article  Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. The Journal of Machine Learning Research, 3, 993–1022.

    MATH  Google Scholar 

  • Bonacich, P. (1972). Factoring and weighting approaches to status scores and clique identification. Journal of Mathematical Sociology, 2(1), 113–120.

    Article  Google Scholar 

  • Borgatti, S. P. (2005). Centrality and network flow. Social Networks, 27(1), 55–71.

    Article  Google Scholar 

  • Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.

    MathSciNet  MATH  Google Scholar 

  • Breunig, M. M., Kriegel, H., Ng, R. T., & Sander, J. (2000). LOF: Identifying density-based local outliers. ACM Sigmod Record, 29(2), 93–104.

    Article  Google Scholar 

  • Buss, A. R. (1976). Evaluation of Canadian psychology departments based upon citation and publication counts. Canadian Psychological Review/Psychologie Canadienne, 17(2), 143.

    Article  Google Scholar 

  • Butler, L. (2005). What happens when funding is linked to publication counts? Handbook of quantitative science and technology research (pp. 389–405). NewYork: Springer.

    Google Scholar 

  • Cowan, R., & Jonard, N. (2003). The dynamics of collective invention. Journal of Economic Behavior & Organization, 52(4), 513–532.

    Article  Google Scholar 

  • Creamer, E. G. (1998). Assessing faculty publication productivity: Issues of equity. ASHE-ERIC higher education report, volume 26, number 2. ERIC.

  • Creamer, E. G. (1998). Assessing faculty publication productivity: Issues of equity. ASHE-ERIC higher education report, volume 26, number 2. ERIC.

  • De Bellis, N. (2009). Bibliometrics and citation analysis: From the science citation index to cybermetrics. Lanham: Scarecrow Press.

    Google Scholar 

  • De Nooy, W., Mrvar, A., & Batagelj, V. (2011). Exploratory social network analysis with Pajek (27th ed.). Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Deng, H., Runger, G., & Tuv, E. (2011). Bias of importance measures for multi-valued attributes and solutions. Artificial neural networks and machine Learning–ICANN 2011 (pp. 293–300) Springer.

  • Diestel, R. (2005). Graph theory, vol. 173 of. Graduate Texts in Mathematics.

  • Dundar, H., & Lewis, D. R. (1998). Determinants of research productivity in higher education. Research in Higher Education, 39(6), 607–631.

    Article  Google Scholar 

  • Ebadi, A., & Schiffauerova, A. (2013). Impact of funding on scientific output and collaboration: A survey of literature. Journal of Information & Knowledge Management, 12(04), 1350037.

    Article  Google Scholar 

  • Ebadi, A., & Schiffauerova, A. (2015a). A computer system for automatic evaluation of researchers’ performance, 15th International Society of Scientometrics and Informetrics Conference (ISSI 2015), 425–435.

  • Ebadi, A., & Schiffauerova, A. (2015b). On the relation between the small world structure and scientific activities. PLoS One, 10(3), e0121129.

    Article  Google Scholar 

  • Ebadi, A., & Schiffauerova, A. (2015c). How to receive more funding for your research? Get connected to the right people! PLoS One, 10(7), e0133061.

    Article  Google Scholar 

  • Ebadi, A., & Schiffauerova, A. (2015d). How to become an important player in scientific collaboration networks? Journal of Informetrics, 9(4), 809–825.

    Article  Google Scholar 

  • Elzinga, A., & Jamison, A. (1995). Changing policy agendas in science and technology. In Sheila Jasanoff (Ed.), Handbook of science and technology studies. London: Sage.

    Google Scholar 

  • Eslami, H., Ebadi, A., & Schiffauerova, A. (2013). Effect of collaboration network structure on knowledge creation and technological performance: The case of biotechnology in Canada. Scientometrics, 97(1), 99–119.

    Article  Google Scholar 

  • Fu, L. D., & Aliferis, C. F. (2010). Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature. Scientometrics, 85(1), 257–270.

    Article  Google Scholar 

  • Fu, L. D., Aphinyanaphongs, Y., & Aliferis, C. F. (2013). Computer models for identifying instrumental citations in the biomedical literature. Scientometrics, 97(3), 871–882.

    Article  Google Scholar 

  • Garfield, E. (1970). Citation indexing for studying science. Essays of an Information Scientist, 1, 133–138.

    Google Scholar 

  • Gingras, Y. (1996). Bibliometric analysis of funded research. A feasibility study. Ottawa: Report to the Program Evaluation Committee of NSERC.

    Google Scholar 

  • Godin, B. (2003). The impact of research grants on the productivity and quality of scientific research. No. 2003. INRS working paper.

  • Gross, P. L. K., & Gross, E. M. (1927). College libraries and chemical education. Science, 66(1713), 385–389.

    Article  Google Scholar 

  • Hanneman, R. A., & Riddle, M. (2011). Concepts and measures for basic network analysis. In J. Scott, & P. J. Carringon (Eds.), The SAGE handbook of social network analysis (pp. 340–367). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Hicks, D., Tomizawa, H., Saitoh, Y., & Kobayashi, S. (2004). Bibliometric techniques in the evaluation of federally funded research in the United States. Research Evaluation, 13(2), 76–86.

    Article  Google Scholar 

  • Huang, J., Zhuang, Z., Li, J., & Giles, C. L. (2008). Collaboration over time: Characterizing and modeling network evolution. Proceedings of the 2008 international conference on web search and data mining, pp. 107–116.

  • Huffman, W. E., & Evenson, R. E. (2005). New econometric evidence on agricultural total factor productivity determinants: Impact of funding composition. Iowa State University, Department of Economics, Working paper, 3029.

  • Jacob, B. A., & Lefgren, L. (2011). The impact of research grant funding on scientific productivity. Journal of Public Economics, 95(9), 1168–1177.

    Article  Google Scholar 

  • Katz, J. S., & Martin, B. R. (1997). What is research collaboration? Research Policy, 26(1), 1–18.

    Article  Google Scholar 

  • King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of Information Science, 13(5), 261–276.

    Article  Google Scholar 

  • Kohavi, R., & Provost, F. (1998). Confusion matrix. Machine Learning, 30(2–3), 271–274.

    Google Scholar 

  • Kyvik, S., & Olsen, T. B. (2008). Does the aging of tenured academic staff affect the research performance of universities? Scientometrics, 76(3), 439–455.

    Article  Google Scholar 

  • Lee, S., & Bozeman, B. (2005). The impact of research collaboration on scientific productivity. Social Studies of Science, 35(5), 673–702.

    Article  Google Scholar 

  • Luukkonen-Gronow, T. (1987). Scientific research evaluation: A review of methods and various contexts of their application. R&D Management, 17(3), 207–221.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1996). Problems of citation analysis. Scientometrics, 36(3), 435–444.

    Article  Google Scholar 

  • Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Byers, A. H. (2011). Big data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

  • McAllister, P. R., & Narin, F. (1983). Characterization of the research papers of US medical schools. Journal of the American Society for Information Science, 34(2), 123–131.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago press.

    Google Scholar 

  • Nicolaisen, J. (2002). The J-shaped distribution of citedness. Journal of Documentation, 58(4), 383–395.

    Article  Google Scholar 

  • Okubo, Y. (1997). Bibliometric indicators and analysis of research systems: Methods and examples (Vol. 1997/1). Paris: OECD Publishing.

    Book  Google Scholar 

  • Payne, A. A., & Siow, A. (2003). Does federal research funding increase university research output? Advances in Economic Analysis & Policy, 3(1), 1–22.

    Article  Google Scholar 

  • Peritz, B. C. (1990). The citation impact of funded and unfunded research in economics. Scientometrics, 19(3–4), 199–206.

    Article  Google Scholar 

  • Phelan, T. (1999). A compendium of issues for citation analysis. Scientometrics, 45(1), 117–136.

    Article  Google Scholar 

  • Plume, A., & van Wiejen, D. (2014). Publish or perish? The rise of the fractional author. Trends Journal of Sciences Research, 38.

  • Polster, C. (2007). The nature and implications of the growing importance of research grants to canadian universities and academics. Higher Education, 53(5), 599–622.

    Article  Google Scholar 

  • Porter, S. R., & Umbach, P. D. (2001). Analyzing faculty workload data using multilevel modeling. Research in Higher Education, 42(2), 171–196.

    Article  Google Scholar 

  • Quinlan, J. R. (1993). C4. 5: Programs for machine learning, Morgan Kaufmann.

  • Sanz Menéndez, L., & Borrás, S. (2000). Explaining changes and continuity in EU technology policy: The politics of ideas.

  • Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.

    Article  Google Scholar 

  • Sonnenwald, D. H. (2007). Scientific collaboration. Annual Review of Information Science and Technology, 41(1), 643–681.

    Article  Google Scholar 

  • Tan, D. L. (1986). The assessment of quality in higher education: A critical review of the literature and research. Research in Higher Education, 24(3), 223–265.

    Article  Google Scholar 

  • Tijssen, R. J. (2004). Is the commercialisation of scientific research affecting the production of public knowledge?: Global trends in the output of corporate research articles. Research Policy, 33(5), 709–733.

    Article  Google Scholar 

  • Van Raan, A. F. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.

    Article  Google Scholar 

  • Wanner, R. A., Lewis, L. S., & Gregorio, D. I. (1981). Research productivity in academia: A comparative study of the sciences, social sciences and humanities. Sociology of Education, 54, 238–253.

    Article  Google Scholar 

  • Wasserman, S. (1994). Social network analysis: Methods and applications. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Watts, D. J., & Strogatz, S. H. (1998). Collective dynamics of ‘small-world’ networks. Nature, 393(6684), 440–442.

    Article  Google Scholar 

  • Weiss, S., & Kulikowski, C. (1991). Computer systems that learn. California: Morgan Kaufmann Publishers.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montreal, QC, Canada

    Ashkan Ebadi & Andrea Schiffauerova

  2. Department of Engineering Systems and Management, Masdar Institute of Science and Technology, Abu Dhabi, United Arab Emirates

    Andrea Schiffauerova

Authors
  1. Ashkan Ebadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Schiffauerova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashkan Ebadi.

Appendix: Sample of prediction results

Appendix: Sample of prediction results

Variables are listed in Table 4 and samples of the predictions for both prediction tasks are presented in Tables 5 and 6. The real value of the target variable is highlighted in light grey where the respective predicted value is highlighted in dark grey.

Table 4 Description of the variables
Full size table
Table 5 Sample of iSEER prediction results, Task P1
Full size table
Table 6 Sample of iSEER prediction results, Task P2
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ebadi, A., Schiffauerova, A. iSEER: an intelligent automatic computer system for scientific evaluation of researchers. Scientometrics 107, 477–498 (2016). https://doi.org/10.1007/s11192-016-1852-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-1852-2

Keywords

Search

Navigation