Skip to main content
SpringerLink
Log in

Citation analysis of research performer quality

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Background: Citation analysis for evaluative purposes typically requires normalization against some control group of similar papers. Selection of this control group is an open question. Objectives: Gain a better understanding of control group requirements for credible normalization. Approach: Performed citation analysis on prior publications of two proposing research units, to help estimate team research quality. Compared citations of each unit"s publications to citations received by thematically and temporally similar papers. Results: Identification of thematically similar papers was very complex and labor intensive, even with relatively few control papers selected. Conclusions: A credible citation analysis for determining performer or team quality should have the following components: – Multiple technical experts to average out individual bias and subjectivity; – A process for comparing performer or team output papers with a normalization base of similar papers; – A process for retrieving a substantial fraction of candidate normalization base papers; Manual evaluation of many candidate normalization base papers to obtain high thematic similarity and statistical representation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Albert, M.B., Avery, D., Narin F., Mcallister P. (1991), Direct validation of citation counts as indicators of industrially important patents, Research Policy, 20 (3): 251-259.

    Google Scholar 

  • Braam, R.R., Moed H.F., Van Raan A.F.J. (1988), Mapping of science by combined co-citation and word analysis. 1. Structural aspects, Science Technology & Human Values, 13 (1–2): 97-98.

    Google Scholar 

  • Braam, R.R., Moed, H.F., Van Raan A.F.J. (1991), Mapping of science by combined cocitation and word analysis. 1. Structural aspects, Journal of the American Society for Information Science, 42 (4): 233-251.

    Google Scholar 

  • Davidse, R. J., Van Raan, A. F. J. (1997), Out of particles: Impact of CERN, DESY, and SLAC research to fields other than physics, Scientometrics, 40 (2): 171-193.

    Google Scholar 

  • DOE, An Assessment of the Basic Energy Sciences Program, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER-0123, March 1982.

  • Festschrift (2000), Web of Knowledge-A Festschrift in Honor of Eugene Garfield, ASIST Monograph Series.

  • Garfield, E. (1955), Citation indexes for science-new dimension in documentation through association of ideas, Science, 122 (3159): 108-111.

    Google Scholar 

  • Garfield, E., Sher, I. H. (1963), New factors in evaluation of scientific literature through citation indexing, American Documentation, 14 (3): 195-201.

    Google Scholar 

  • Garfield, E. (1964), Science Citation Index-new dimension in indexing, Science, 144 (361): 649-654.

    Google Scholar 

  • Garfield, E. (1965), Can citation indexing be automated, Statistical Association Methods For Mechanized Documentation Symposium Proceedings, 1964: (Nbs26) 189.

    Google Scholar 

  • Garfield, E. (1966), Patent citation indexing and notions of novelty similarity and relevance, Journal of Chemical Documentation 6 (2): 63-65.

    Google Scholar 

  • Garfield, E. (1970), Citation indexing for studying science, Nature, 227 (5259): 669-671.

    Google Scholar 

  • Geisler, E. (2000), The Metrics of Science and Technology, Quorum Books, Westport, CT.

    Google Scholar 

  • Kostoff, R. N. (1997a), The Handbook of Research Impact Assessment, Seventh Edition, Summer 1997, DTIC Report Number ADA296021. Also, available at http://www.dtic.mil/dtic/kostoff/index.html

  • Kostoff, R. N. (1997b), Peer review: The appropriate GPRA metric for research, Science, 277 (5326): 651-652.

    Google Scholar 

  • Kostoff, R. N. (1997c), Citation analysis cross-field normalization: A new paradigm, Scientometrics, 39 (3): 225-230.

    Google Scholar 

  • Kostoff, R. N., Eberhart, H. J., Toothman, D. R. (1997d), Database tomography for information retrieval, Journal of Information Science, 23 (4) 301-311.

    Google Scholar 

  • Kostoff, R. N. (1998), The use and misuse of citation analysis in research evaluation, Scientometrics, 43 (1): 27-43.

    Google Scholar 

  • Kostoff, R. N. (2000a), The underpublishing of science and technology results, The Scientist, 14 (9): 6.

    Google Scholar 

  • Kostoff, R. N. (2000b), High quality information retrieval for improving the conduct and management of research and development, Proceedings: Twelfth International Symposium on Methodologies for Intelligent Systems, 11–14 October.

  • Kostoff, R. N., Hartley, J. (2001a), Structured abstracts for technical journals, Science, 292 (5519): 1067.

    Google Scholar 

  • Kostoff, R. N., Hartley, J. (2000b), Structured abstracts for technical journal articles, Letter to Technical Journal Editors, 14 May. Letter available from author.

  • Macroberts, M.H., Macroberts, B.R. (1989), Problems of citation analysis: A critical review, Journal of the American Society for Information Science, 40 (5): 342-349.

    Google Scholar 

  • Macroberts, M., Macroberts, B. (1996), Problems of citation analysis, Scientometrics, 36 (3): 435-444.

    Google Scholar 

  • Marshakova, I.V. (1973), System of document connections based on references, Nauchno-Tekhnicheskaya Informatsiya Seriya 2-Informatsionnye Protsessy I Sistemy, (6) 3-8.

  • Marshakova, I.V. (1981), Citation networks in information-science, Scientometrics, 3 (1): 13-25.

    Google Scholar 

  • Marshakova, I.V. (1988), On the mapping of science, Vestnik Akademii Nauk SSSR, (5) 70-82.

  • Martin, B. Irvine, J. (1983), Assessing basic research: some partial indicators of scientific progress in radio-astronomy, Research Policy, 12: 61-69.

    Google Scholar 

  • Moed, H.F., Van Raan, A.F.J. (1986), Observations and hypotheses on the phenomenon of multiple citation to a research groups oeuvre, Scientometrics, 10 (1–2): 17-33.

    Google Scholar 

  • Narin, F., Carpenter, M.P. (1975), National publication and citation comparisons, Journal of the American Society for Information Science, 26 (2): 80-93.

    Google Scholar 

  • Narin, F. (1976), Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity, (monograph), NSF C-637, National Science Foundation, Contract NSF C-627, NTIS Accession No. PB252339/AS, March 31.

  • Narin, F., Carpenter, M.P., Woolf, P. (1984), Technological performance assessments based on patents and patent citations, IEEE Transactions On Engineering Management, 31 (4): 172-183.

    Google Scholar 

  • Narin, F., Olivastro, D., Stevens, K. A. (1994), Bibliometrics-theory, practice, and problems, In: Kostoff, R. N., (Ed.) Evaluation Review, Special Issue on Research Impact Assessment, 18: 1.

  • Narin, F., Hamilton, K.S. (1996), Bibliometric performance measures, Scientometrics, 36 (3): 293-310.

    Google Scholar 

  • Nederhof, A.J., Van Raan, A.F.J. (1987), Citation theory and the Ortega hypothesis, Scientometrics, 12 (5–6): 325-328.

    Google Scholar 

  • Schubert, A., GlÄnzel, W., Braun, T. (1986), Relative indicators of publication output and citation impact of European physics research: 1978–1980, Czechoslovak Journal of Physics, 36 (1): 126-129.

    Google Scholar 

  • Schubert, A., Braun, T. (1993), Reference-standards for citation based assessments, Scientometrics, 26 (1): 21-35.

    Google Scholar 

  • Schubert, A., Braun, T. (1996), Cross-field normalization of scientometric indicators, Scientometrics, 36 (3): 311-324.

    Google Scholar 

  • Small, H. G. (1973), Relationship between citation indexing and word indexing-Study of co-occurrences of title words and cited references, Proceedings of the American Society for Information Science, 10: 217-218.

    Google Scholar 

  • Small, H. (1974), Co-citation in scientific literature-New measure of relationship between 2 documents, Current Contents (7): 7-10.

    Google Scholar 

  • Small, H. G. (1977), Co-citation model of a scientific specialty-Longitudinal-study of collagen research, Social Studies of Science, 7 (2): 139-166.

    Google Scholar 

  • Small, H. (1981), The relationship of information-science to the social-sciences-A co-citation analysis, Information Processing & Management, 17 (1): 39-50.

    Google Scholar 

  • Small, H., Sweeney, E., GREENLEE, E. (1985), Clustering the science citation index using co-citations. 2. Mapping science, Scientometrics, 8 (5–6): 321-340.

    Google Scholar 

  • Sullivan, D., White, D. H., Barboni, E. J. (1977) Co-citation analyses of science-Evaluation, Social Studies of Science, 7 (2): 223-240.

    Google Scholar 

  • Sullivan, D., Koester, D., White, D. H., Kern, R. (1979), Understanding rapid theoretical change in particle physics-Month-by-month co-citation analysis, Proceedings of the American Society for Information Science, 16: 276-285.

    Google Scholar 

  • Sullivan, D., Koester, D., White, D. H., Kern, R. (1980), Understanding rapid theoretical change in particle physics-A month-by-month co-citation analysis, Scientometrics, 2 (4): 309-319.

    Google Scholar 

  • Van Raan, A.F.J. (1991), Fractal geometry of information space as represented by co-citation-clustering, Scientometrics, 20 (3): 439-449.

    Google Scholar 

  • Van Raan, A.F.J., Tijssen R.J.W. (1993), The neural net of neural network research-An exercise in bibliometric mapping, Scientometrics, 26 (1): 169-192.

    Google Scholar 

  • Van Raan, A.F.J. (1996), Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises, Scientometrics, 36 (3): 397-420.

    Google Scholar 

  • Zachlin, A. C. (1948), On literature citation, Science, 107 (2777): 292-293.

    Google Scholar 

  • Zirkle, C. (1954), Citation of fraudulent data, Science, 120 (3109): 189-190.

    Google Scholar 

  • Zsindely, S., Schubert A., Braun T. (1982), Citation patterns of editorial gatekeepers in international chemistry journals, Scientometrics, 4 (1): 69-76.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Office of Naval Research, Arlington 800 N. Quincy St, USA

    Ronald N. Kostoff

Authors
  1. Ronald N. Kostoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kostoff, R.N. Citation analysis of research performer quality. Scientometrics 53, 49–71 (2002). https://doi.org/10.1023/A:1014831920172

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1014831920172

Keywords

Search

Navigation