Abstract
Although there are several methods for determining the quality of scientific research, there is no satisfactory method known that can measure the utilization of it. Earlier proposed methods measure a particular kind of utilization, but are — in practice — a poor indication for the utilization on the whole, a concept for which a definition is hard to make. These methods do not comply with the construct validity. The main problem in this case is the great diversity of what we mean by use of results of scientific research, resulting in a lack of consensus on the criteria for assessing the utilization. Here, we propose and discuss two methods. To evaluate utilization in a broad sense the four-dimensional model describes the degree of utilization with three, mostly independent, aspects: the involvement of the user, the availability of a transferable research product, and the commercial benefits resulting from the research results. In the other method the utilization of the research results is described first, and subsequently the utilization is quantified by a jury, who group the different projects in five classes, based on a Guttman scale.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
P. S. Nagpaul, Quasi-quantitative measures of research performance: an assessment of construct validity and reliability,Scientometrics, 33 (1995) 169–185.
M. Nyden, Evaluation of R&D in Sweden, In:J. Mayne (Ed.),Advancing Public Policy Evaluation, Elsevier Science Publishers, 1992, p. 159–171.
M. J. Ledoux, L. Bach, N. Conde-Molist, M. Matt, V. Schaeffer, Evaluation of the economic effects of Brite/Euram programmes on the European industry. Presentation at theFirst International Conference on Research Evaluation in Thesaloniki, 1995.
V. Stolte-Heiskanen, Trends and problems in the evaluation of research, In:E. Kaukonen, V. Stolte-Heiskanen (Eds),Science Studies and Science Policy: Proceedings of a Finnish Bulgarian Symposium, Academy of Finland (Coordination Research Group for Science of Science Studies) and University of Tampere, Finland, 1984.
D. Andrich,Rasch Models for Measurement, Sage publications Inc., Murdoch university Australia, 1988.
H. D. Daniel, R. Fisch, Research performance evaluation in the German university sector,Scientometrics, 19 (1990) 349–362.
A. Frenkel, T. Reiss, S. Maital, K. Koschatzky, H. Grupp, Technometric evaluation and technology policy: the case of biodiagnostics kits in Israel,Research policy, 23 (1994) 281–292.
R. E. Berger, P. J. Saaverda, Commercialization activities in the SBIR Program (part 1)The Journal of Technology Transfer, Fall 17 (1992) no. 4, 27–39.
F. Narin, D. Olivastro, The technological utilization of European science, Presentation at theJoint EC-Leiden Conference on Science & Technology Indicators in Leiden, the Netherlands. 1991.
B. Bozeman, J. Melkers (Eds),Evaluating R&D Impacts: Methods and Practice, Kluwer Academic Publishers, Boston, 1993.
A. J. Nederhof, R. F. Meijer, Development of bibliometric indicators for utility of research to users in society: measurement of external knowledge transfer via publication in trade journals,Scientometrics, 32 (1995) 37–48.
E. C. Noyons, A. F. J. van Raan, Bibliometric cartography of scientific and technological developments of a R&D field,Scientometrics, 30 (1994) 157–173.
W. P. van Els, C. N. M. Jansz, C. le Pair, Formal evaluation methods: their utility and limitations,Scientometrics, 17 (1989) 415–425.
F. C. H. D. van den Beemt, A. F. J. van Raan, Evaluating research proposals,Nature, 375 (1995) 272.
M. A. Brown, T. R. Curlee, S. R. Elliott, Evaluating technology innovation programs: the use of comparison groups to identify impacts,Research Policy, 24 (1995) 669–684.
K. S. Cameron, Measuring organizational effectiveness in institutions of higher education,Administrative Science Quarterly, 23 (1978) 604–632.
C. A. M. Mombers, C. le Pair, The predictive value of peer review for the outcome of technological projects — a users' view. Presentation atThe 18th Annual Meeting of the Society for Social Studies of Science (4S) in Lafayette, In., USA, 1993.
K. Alewell, Criteria for performance profiles of departments and universities,Scientometrics, 19 (1990) 337–347.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
van Caulil, G.F., Mombers, C.A.M. & van den Beemt, F.C.H.D. Quantifying the utilization of research: The difficulties and two models to evaluate the utilization of research results. Scientometrics 37, 433–444 (1996). https://doi.org/10.1007/BF02019257
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF02019257