Abstract
This paper argues that research performance is essentially a multidimensional concept which cannot be encapsulated into a single universal criterion. Various indicators used in quantitative studies on research performance at micro or meso-levels can be classified into two broad categories: (i) objective or quantitative indicators (e.g. counts of publications, patents, algorithms or other artifacts of research output) and (ii) subjective or qualitative indicators which represent evaluative judgement of peers, usually measured on Likert or semantic differential scales. Because of their weak measurement properties, subjective indicators can also be designated as quasi-quantitative measures. This paper is concerned with the factorial structure and construct validity of quasi-quantitative measures of research performance used in a large-scale empirical study carried out in India. In this study, a reflective measurement model incorporating four latent variables (R & D effectiveness, Recognition, User-oriented effectiveness and Administrative effectiveness) is assumed. The latent variables are operationalized through thirteen indicators measured on 5-point semantic differential scales. Convergent validity, discriminant validity and reliability of the measurement model are tested through LISREL procedure.
Similar content being viewed by others
Notes and references
See for example,P. S. Nagpaul, Conceptual and methodological problems in the measurement of research performance, In:P. S. Nagpaul (Ed.),Organization and Efficiency of Research Groups, National Institute of Science, Technology and Development Studies, New Delhi, 1987.
M. R. Moser, Measuring R & D performance,Research Management, 5 (1985) 31–33.
K. S., Cameron, Measuring organizational effectiveness in institutions of higher education,Administrative Science Quarterly, 23 (1978) 604–632.
F. M. Andrews (Ed.),Scientific Productivity, Cambridge University Press, 1979.
P. H. Birnbaum, Predictors of long term research performance, In:S. R. Epton, et al (Eds)Managing Interdisciplinary Research, John Wiley & Sons, New York, 1974;P. S. Nagpaul,op. cit. Ref. 1. Conceptual and methodological problems in the measurement of research performance, In:P. S. Nagpaul (Ed.),Organization and Efficiency of Research Groups, National Institute of Science, Technology and Development Studies, New Delhi, 1987.
G. Cole, Classifying research units by performance and influence: A typology of Round I data, In:F. M. Andrews, (Ed.),op. cit. Scientific Productivity, Cambridge University Press, 1979. Ref. 4, 353–404; M. F. Fox, Publication productivity among scientists: A critical review,Social Studies of Science, 13 (1983) 285–305.
G. Cole,op. cit. Ref. 6 Classifying research units by performance and influence: A typology of Round I data, In:F. M. Andrews, (Ed.),op. cit. Scientific Productivity, Cambridge University Press, 1979. Ref. 4, 353–404; M. F. Fox, Publication productivity among scientists: A critical review,Social Studies of Science, 13 (1983) 285–305.
V. Stolte-Heiskanen, Trends and problems in the evaluation of research, In:E. Kaukonen, V. Stolte-Heiskanen (Eds),Science Studies and Science Policy: Proceedings of a Finnish Bulgarian Symposium, Academy of Finland (Coordination Research Group for Science of Science Studies) and University of Tampere, Finland, 1984.
P. S. Napaul,op. cit. Ref. 1..
L. Carter, A new look at NSF,Science, 204 (1979) 1064–1065.
V. Stolte-Heiskanen,op. cit. Ref. 8..
In this study, a research unit is operationally defined as a unit that has at least one project, has a total expected life span of at least one year and comprises at least four core members among whom there is one scientist who is the head of the unit.
A. H. Vande Ven, A. L. Delbeg, K. Koenig, Jr, Determinants of coordination modes within organizations,American Sociological Review, 41 (1976) 322–338.
S. Katz, N. L. Johnson (chief Eds)Encyclopedia of Statistical Sciences, Vol. 1 (1982) 189–191.
D. T. Campbell, D. W. Fiske, Convergent and discriminant validation by the multitrait-multimethod matrix,Psychological Bulletin, 56 (1959) 81–105.
D. T. Campbell, D. W. Fiske,op. cit. Ref. 15..
R. P. Bagozzi, L. W. Phillips, Representing and testing organizational theories: A holistic construct,Administrative Science Quarterly, 27 (1980) 459–489.
L. J. Cronbach, Test validation, In:R. L. Thorndike (Ed.)Educational Measurement, 2nd ed., American Council on Education, Washington, DC.
F. M. Lord, M. R. Novick,Statistical Theories of Mental Test Scores, Addison Wesley, Reading, MA, 1968.
N. Venkatraman, Strategic orientation of business enterprises: the constructs, dimensionality and measurement,Management Science, 11 (1989) 942–962.
C. E. Werts, R. L. Linn, K. G. Joreskog, Interclass reliability estimates: testing structural assumptions,Educational and Psychological Measurement, 34 (1974) 25–33.
D. D. Duncan,Introduction to Structural Equation Models, Academic Press, New York, 1975, p. 113.
S. K. Reddy, Effects of ignoring correlated measurement errors in structural equation models,Educational and Psychological Measurement, 52 (1992) 549–570.
J. C. Anderson, D. W. Gerbing, The effect of sampling error on convergence, improper solutions and goodness of fit indices for maximum likelihood confirmatory factor analysis,Psychometrika, 49 (1984) 155–173;R. P. Bagozzi, Issues in the application of covariance structure analysis: a further comment,Journal of Consumer Research, 9 (1983) 449–450;C. Fornell, Issues in the application of covariance structure analysis; a comment,Journal of Consumer Research, 9 (1983) 443—447.
S. K. Reddy,op. cit. Ref. 23..
R. P. Bagozzi, A holistic methodology for modeling consumer response to innovation,Operations Research, 31 (1983) 128–176.
K. Joreskog, D. Sorbom,LISREL 7.16: Analysis of Linear Structured Relationship by Maximum Likelihood and Least Squares Method, International Educational Services, Chicago, 1989.
D. N. Lawley, A. E. Maxwell,Factor Analysis as a Statistical Method, Butterworth, London, 1971.
M. W. Browne, R. Cuddeck, Alternative ways of assessing model fit,Sociological Methods and Research, 21 (1992) 230–258.
K. A. Bollen, A new incremental fit index for general structural equation models,Psychometrika, 51 (1992) 335–373.
S. Mulaik, L. James, J. Van Alstine, N. Bennett, S. Lino, C. Stilwell, Evaluation of goodness-of-fit indices for structural equation models,Psychological Bulletin, 105 (1987) 430–445.
P. M. Bentler, D. G. Bonnet, Significance tests and goodness of fit in the analysis of covariance structures,Psychological Bulletin, 88 (1980) 588–606.
P. M. Bentler, D. G. Bonnet,op. cit. Ref. 32..
K. A. Bollen,op. cit. Ref. 30..
R. K. Parkes, Field dependence and the differentiation of neurotic syndromes, In:P. Cuttanece, R. Ecob (Eds),Structural Modeling by Example, Cambridge University Press, Cambridge, 1987.
See for example.J. F. A. Spangenberg, W. Buijink, W. Alfennar, Some incentives and constraints of scientific performance in departments of economics: Part I. Predictor criterion relations,Scientometrics, 18 (1990) 241–268;F. M. Andrews,op. cit. Ref. 4.Scientific Productivity, Cambridge University Press, 1979.
H. Morita-Lou,Science and Technology Indicators for Development, United Nations Science and Technology for Development Series, Westview Press, London, 1985.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Nagpaul, P.S. Quasi-quantitative measures of research performance: An assessment of construct validity and reliability. Scientometrics 33, 169–185 (1995). https://doi.org/10.1007/BF02020567
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF02020567