Abstract
This paper discusses a concept for inferring attributes of ‘frontier research’ in peer-reviewed research proposals under the popular scheme of the European Research Council (ERC). The concept serves two purposes: firstly to conceptualize, define and operationalize in scientometric terms attributes of frontier research; and secondly to build and compare outcomes of a statistical model with the review decision in order to obtain further insight and reflect upon the influence of frontier research in the peer-review process. To this end, indicators across scientific disciplines and in accord with the strategic definition of frontier research by the ERC are elaborated, exploiting textual proposal information and other scientometric data of grant applicants. Subsequently, a suitable model is formulated to measure ex-post the influence of attributes of frontier research on the decision probability of a proposal to be accepted. We present first empirical data as proof of concept for inferring frontier research in grant proposals. Ultimately the concept is aiming at advancing the methodology to deliver signals for monitoring the effectiveness of peer-review processes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The indicators are introduced and discussed in the following. Here they are firstly named.
PE (LS) holds ten (nine) main and ~170 (100) subcategories. The third domain “Social Sciences & Humanities” is not considered as it is expected to differ in terms of publishing, citation behaviour, and other features from those observed in PE and LS (e.g., national/regional orientation, less publications in form of articles, different theoretical ‘development rate’, number of authors, non-scholarly publications), which make it less assessable for approaches developed for the natural and life sciences (Nederhof 2006; Juznic et al. 2010).
References
Adam, D. (2002). Citation analysis: The counting house. Nature, 415, 726–729.
Antonoyianakis, M., Hemmelskamp, J., & Kafatos, F. C. (2009). The European research council takes flight. Cell, 136, 805–809.
van den Besselaar, P., & Leydesdorff, L. (2009). Past performance, peer review and project selection: A case study in the social and behavioural sciences. Research Evaluation, 18, 273–288.
Bornmann, L., & Daniel, H. D. (2008). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178.
Boyack, K. W., & Klavans, R. (2010). Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately? Journal of the American Society for Information Science (JASIST), 61(12), 2389–2404.
Chen, C. (2005). Measuring the movement of a research paradigm. In Proceedings of SPIE-IS&T, Visualization and data analysis. V 5669, (pp. 63–76). San Jose.
Czerwon, H. J., & Glänzel, W. (1995). A new methodological approach to bibliographic coupling and its application to research-front and other core documents. Proceedings of the 5th international conference on scientometrics and informetrics, (pp. 7–10). River Forrest, Illinois, US.
EC (2005). Frontier research: The European challenge high level expert group report. European commission.
Egghe, L. (2006). Theory and practice of the g-index. Scientometrics, 69(1), 131–152.
ERC (2008). ERC Work Programme 2009.
Glänzel, W., & Schoepflin, U. (1995). A bibliometric study on ageing and reception processes of scientific literature in the science. Journal of Information Science, 21(1), 37–53.
Gorraiz, J., & Schiebel, E. (Eds.) (2008). Excellence and emergence—a new challenge for the combination of quantitative and qualitative approaches. In Book of abstracts of the 10th international conference on science and technology indicators Vienna, Austria.
Greene, W. H. (2003). Econometric analysis (5th ed.). Upper Saddle River: Prentice Hall.
Gupta, B. M. (1997). Analysis of distribution of the age of citations in theoretical population genetics. Scientometrics, 40(1), 139–162.
Han, J., & Kamber, M. (2001). Data mining: Concepts and techniques. San Francisco: Morgan Kaufmann Publishers.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of data mining. Cambridge: The MIT Press.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. PNAS, 102(46), 16569–16572.
Hojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003). Impartial judgment by the “gatekeepers” of science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8(1), 75–96.
Hörlesberger, M., Holste, D., Schiebel, E., Roche, I., François, C., Besagni, D., et al. (2011). Measuring the preferences of the scientific orientation of authors from their profiles of cited references. Rome: ENID (European Network of Indicator Designers).
Jin, B. (2006). H-index: An evaluation indicator proposed by scientist. Science Focus, 1(1), 8–9.
Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R- and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.
Juznic, P., Peclin, S., Zaucer, M., Mandelj, T., Pusnik, M., & Demsar, F. (2010). Scientometric indicators: Peer-review, bibliometric methods and conflict of interest. Scientometrics, 85, 429–441.
Klavans, R., & Boyack, K. W. (2006). Quantitative evaluation of large maps of science. Scientometrics, 68, 475–499.
Klavans, R., & Boyack, K. W. (2008). U.S. Vulnerabilities in science and engineering. In 10th international conference on Science and Technology Indicators (S&TI), Vienne, 17–20 September 2008, (pp. 86–88).
Lelu, A. (1993). Modèles neuronaux pour l’analyse de données documentaires et textuelles. PhD Dissertation: Université de Paris. 6.
Lelu, A., & François, C. (1992). Hypertext paradigm in the field of information retrieval: A neural approach. In 4th ACM conference on hypertext, Milano, 30th November–4th December.
Mahgoub, H., Rösner, D., Ismail, N., & Torkey, F. (2008). A text mining technique using association rules. International Journal of Computational Intelligence, 4(1), 2008.
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications—reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160–168.
Nederhof, A. J. (2006). Bibliometric monitoring of research performance in the social sciences and the humanities: A review. Scientometrics, 66, 81–100.
van Noorden, R. (2010). A profusion of measures. Nature, 465, 864–865.
Polanco, X., François, C., Royaute, J., Besagni, D., Roche, I. (2001). STANALYST: An integrated environment for clustering and mapping analysis on science and technology, In 8th international conference on scientometrics and informetrics, Proceedings Vol. 2, Sydney, Australia, July 16–20, 2001, (pp. 871–873).
Roche, I., Besagni, D., François, C., Hörlesberger, M., Schiebel, E. (2008). Identification and characterisation of technological topics in the field of molecular biology, Proceedings of the 10th international conference on Science and Technology Indicators (S&T I), Vienna, 17-20 September 2008, (pp. 320–322).
Roche, I., Ghribi, M., Vedovotto, N., François, C., Besagni, D., Cuxac, P., Holste, D., Hörlesberger, M., Schiebel, E. (2011). Detecting domain dynamics: Association rule extraction and diachronic clustering techniques in support of expertise, 1st global TechMining conference, Atlanta, USA, 14th September 2011.
Schiebel, E., & Hörlesberger, M. (2007). About the identification of technology specific keywords in emerging technologies: The case of “Magnetoelectronic”, In 11th ISSI conference, Madrid, Spain, 25–27 June 2007.
Schiebel, E., Hörlesberger, M., Roche, I., François, C., & Besagni, D. (2010). An advanced diffusion model to identify emergent research issues: the case of optoelectronic devices. Scientometrics,. doi:10.1007/s11192-009-0137-4.
Shibata, N., Kajikawa, Y., Takeda, Y., & Matsushima, K. (2009). Comparative study on methods of detecting research fronts using different types of citation. Journal of the American Society for Information Science and Technology, V60, 571–580.
Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science (JASIST), 24(4), 265–269.
Srinivas, S., & Viljamaa, K. (2007). Emergence of Economic Institutions: Analysing the third role of universities in Turku, Finland. Regional Studies, 42(3), 323–341.
STOKES D. (1997). Pasteur’s Quadrant, The Brookings Institution.
Sweitzer, B. J., & Cullen, D. J. (1994). How well does a journal’s peer review process function? Journal of the American Medical Association, 272, 152–153.
Train, K. E. (2009). Discrete choice methods with simulation (2nd ed.). Cambridge: Cambridge University Press.
Yoon, B., Lee, S., & Lee, G. (2010). Development and application of a keyword-based knowledge map for effective research. Scientometrics, 85, 803–820.
Zitt, M., & Bassecoulard, E. (2008). Challenges for scientometric indicators: data demining, knowledge-flow measurements and diversity issues. Ethics in Science and Environmental Politics, 8, 49–60.
Acknowledgments
The authors acknowledge the support that this work was partially funded by the Ideas specific programme of the EU’s FP7 Framework Programme for Research and Technological Development (Project Reference No. 240765). The authors thank Helga Nowotny, Jens Hemmelskamp and Ulike Kainz-Fernandez of the ERC for stimulating discussions.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hörlesberger, M., Roche, I., Besagni, D. et al. A concept for inferring ‘frontier research’ in grant proposals. Scientometrics 97, 129–148 (2013). https://doi.org/10.1007/s11192-013-1008-6
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-013-1008-6