Skip to main content
SpringerLink
Log in

Quasi-quantitative measures of research performance: An assessment of construct validity and reliability

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper argues that research performance is essentially a multidimensional concept which cannot be encapsulated into a single universal criterion. Various indicators used in quantitative studies on research performance at micro or meso-levels can be classified into two broad categories: (i) objective or quantitative indicators (e.g. counts of publications, patents, algorithms or other artifacts of research output) and (ii) subjective or qualitative indicators which represent evaluative judgement of peers, usually measured on Likert or semantic differential scales. Because of their weak measurement properties, subjective indicators can also be designated as quasi-quantitative measures. This paper is concerned with the factorial structure and construct validity of quasi-quantitative measures of research performance used in a large-scale empirical study carried out in India. In this study, a reflective measurement model incorporating four latent variables (R & D effectiveness, Recognition, User-oriented effectiveness and Administrative effectiveness) is assumed. The latent variables are operationalized through thirteen indicators measured on 5-point semantic differential scales. Convergent validity, discriminant validity and reliability of the measurement model are tested through LISREL procedure.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes and references

  1. See for example,P. S. Nagpaul, Conceptual and methodological problems in the measurement of research performance, In:P. S. Nagpaul (Ed.),Organization and Efficiency of Research Groups, National Institute of Science, Technology and Development Studies, New Delhi, 1987.

    Google Scholar 

  2. M. R. Moser, Measuring R & D performance,Research Management, 5 (1985) 31–33.

    Google Scholar 

  3. K. S., Cameron, Measuring organizational effectiveness in institutions of higher education,Administrative Science Quarterly, 23 (1978) 604–632.

    Google Scholar 

  4. F. M. Andrews (Ed.),Scientific Productivity, Cambridge University Press, 1979.

  5. P. H. Birnbaum, Predictors of long term research performance, In:S. R. Epton, et al (Eds)Managing Interdisciplinary Research, John Wiley & Sons, New York, 1974;P. S. Nagpaul,op. cit. Ref. 1. Conceptual and methodological problems in the measurement of research performance, In:P. S. Nagpaul (Ed.),Organization and Efficiency of Research Groups, National Institute of Science, Technology and Development Studies, New Delhi, 1987.

    Google Scholar 

  6. G. Cole, Classifying research units by performance and influence: A typology of Round I data, In:F. M. Andrews, (Ed.),op. cit. Scientific Productivity, Cambridge University Press, 1979. Ref. 4, 353–404; M. F. Fox, Publication productivity among scientists: A critical review,Social Studies of Science, 13 (1983) 285–305.

    Google Scholar 

  7. G. Cole,op. cit. Ref. 6 Classifying research units by performance and influence: A typology of Round I data, In:F. M. Andrews, (Ed.),op. cit. Scientific Productivity, Cambridge University Press, 1979. Ref. 4, 353–404; M. F. Fox, Publication productivity among scientists: A critical review,Social Studies of Science, 13 (1983) 285–305.

    Google Scholar 

  8. V. Stolte-Heiskanen, Trends and problems in the evaluation of research, In:E. Kaukonen, V. Stolte-Heiskanen (Eds),Science Studies and Science Policy: Proceedings of a Finnish Bulgarian Symposium, Academy of Finland (Coordination Research Group for Science of Science Studies) and University of Tampere, Finland, 1984.

    Google Scholar 

  9. P. S. Napaul,op. cit. Ref. 1..

    Google Scholar 

  10. L. Carter, A new look at NSF,Science, 204 (1979) 1064–1065.

    Google Scholar 

  11. V. Stolte-Heiskanen,op. cit. Ref. 8..

    Google Scholar 

  12. In this study, a research unit is operationally defined as a unit that has at least one project, has a total expected life span of at least one year and comprises at least four core members among whom there is one scientist who is the head of the unit.

  13. A. H. Vande Ven, A. L. Delbeg, K. Koenig, Jr, Determinants of coordination modes within organizations,American Sociological Review, 41 (1976) 322–338.

    Google Scholar 

  14. S. Katz, N. L. Johnson (chief Eds)Encyclopedia of Statistical Sciences, Vol. 1 (1982) 189–191.

  15. D. T. Campbell, D. W. Fiske, Convergent and discriminant validation by the multitrait-multimethod matrix,Psychological Bulletin, 56 (1959) 81–105.

    PubMed  Google Scholar 

  16. D. T. Campbell, D. W. Fiske,op. cit. Ref. 15..

    PubMed  Google Scholar 

  17. R. P. Bagozzi, L. W. Phillips, Representing and testing organizational theories: A holistic construct,Administrative Science Quarterly, 27 (1980) 459–489.

    Google Scholar 

  18. L. J. Cronbach, Test validation, In:R. L. Thorndike (Ed.)Educational Measurement, 2nd ed., American Council on Education, Washington, DC.

  19. F. M. Lord, M. R. Novick,Statistical Theories of Mental Test Scores, Addison Wesley, Reading, MA, 1968.

    Google Scholar 

  20. N. Venkatraman, Strategic orientation of business enterprises: the constructs, dimensionality and measurement,Management Science, 11 (1989) 942–962.

    Google Scholar 

  21. C. E. Werts, R. L. Linn, K. G. Joreskog, Interclass reliability estimates: testing structural assumptions,Educational and Psychological Measurement, 34 (1974) 25–33.

    Google Scholar 

  22. D. D. Duncan,Introduction to Structural Equation Models, Academic Press, New York, 1975, p. 113.

    Google Scholar 

  23. S. K. Reddy, Effects of ignoring correlated measurement errors in structural equation models,Educational and Psychological Measurement, 52 (1992) 549–570.

    Google Scholar 

  24. J. C. Anderson, D. W. Gerbing, The effect of sampling error on convergence, improper solutions and goodness of fit indices for maximum likelihood confirmatory factor analysis,Psychometrika, 49 (1984) 155–173;R. P. Bagozzi, Issues in the application of covariance structure analysis: a further comment,Journal of Consumer Research, 9 (1983) 449–450;C. Fornell, Issues in the application of covariance structure analysis; a comment,Journal of Consumer Research, 9 (1983) 443—447.

    Google Scholar 

  25. S. K. Reddy,op. cit. Ref. 23..

    Google Scholar 

  26. R. P. Bagozzi, A holistic methodology for modeling consumer response to innovation,Operations Research, 31 (1983) 128–176.

    Google Scholar 

  27. K. Joreskog, D. Sorbom,LISREL 7.16: Analysis of Linear Structured Relationship by Maximum Likelihood and Least Squares Method, International Educational Services, Chicago, 1989.

    Google Scholar 

  28. D. N. Lawley, A. E. Maxwell,Factor Analysis as a Statistical Method, Butterworth, London, 1971.

    Google Scholar 

  29. M. W. Browne, R. Cuddeck, Alternative ways of assessing model fit,Sociological Methods and Research, 21 (1992) 230–258.

    Google Scholar 

  30. K. A. Bollen, A new incremental fit index for general structural equation models,Psychometrika, 51 (1992) 335–373.

    Google Scholar 

  31. S. Mulaik, L. James, J. Van Alstine, N. Bennett, S. Lino, C. Stilwell, Evaluation of goodness-of-fit indices for structural equation models,Psychological Bulletin, 105 (1987) 430–445.

    Google Scholar 

  32. P. M. Bentler, D. G. Bonnet, Significance tests and goodness of fit in the analysis of covariance structures,Psychological Bulletin, 88 (1980) 588–606.

    Google Scholar 

  33. P. M. Bentler, D. G. Bonnet,op. cit. Ref. 32..

    Google Scholar 

  34. K. A. Bollen,op. cit. Ref. 30..

    Google Scholar 

  35. R. K. Parkes, Field dependence and the differentiation of neurotic syndromes, In:P. Cuttanece, R. Ecob (Eds),Structural Modeling by Example, Cambridge University Press, Cambridge, 1987.

    Google Scholar 

  36. See for example.J. F. A. Spangenberg, W. Buijink, W. Alfennar, Some incentives and constraints of scientific performance in departments of economics: Part I. Predictor criterion relations,Scientometrics, 18 (1990) 241–268;F. M. Andrews,op. cit. Ref. 4.Scientific Productivity, Cambridge University Press, 1979.

    Google Scholar 

  37. H. Morita-Lou,Science and Technology Indicators for Development, United Nations Science and Technology for Development Series, Westview Press, London, 1985.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Institute of Science, Technology and Development Studies, Dr. K. S. Krishnan Marg, 110 012, New Delhi, (India)

    P. S. Nagpaul

Authors
  1. P. S. Nagpaul
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nagpaul, P.S. Quasi-quantitative measures of research performance: An assessment of construct validity and reliability. Scientometrics 33, 169–185 (1995). https://doi.org/10.1007/BF02020567

Download citation

  • Received:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02020567

Keywords

Search

Navigation