Skip to main content
Log in

Journal-to-journal citation data: Issues of validity and reliability

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Citation analysis is a useful method for studying a wide range of topics in bibliometrics and the sociology of science. However, many challenges have been made to the validity and reliability of the underlying assumptions, the data, and the methods used in citation studies. This article addresses these issues in three parts. First is a brief review of validity and reliability issues in citation research. Next we explore measurement error in a principal source of journal-to-journal citation data, the Institute for Scientific Information'sJournal Citation Reports. Possible sources of measurement error include discrepancies between citing and cited data, changed or deleted journal titles, aberrant abbreviations, and listing algorithms. The last section is a detailed description of ways to overcome some of the measurement errors. The data and examples are drawn from a journal-to-journal citation study in the fields of Communication, Information Science, and Library Science.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • J. BENIGER (1988). Information and communication: The new convergence.Communication Research, 15 (2), 198–218.

    Google Scholar 

  • R. W. BOHANNON (1986). Letter to the editor.Physical Therapy, 66(9), 1431–1432.

    Google Scholar 

  • C. L. BORGMAN & J. SCHEMENT (1988). Information science and communication science: An essay on convergence. In: PRENTICE, A. (Ed.),Information science as a discipline, (in press) NY: Schuman.

    Google Scholar 

  • B. BOYCE & C. BANNING (1979). Data accuracy in citation studies. RQ, 18, 349–150.

    Google Scholar 

  • R. BROADUS (1987). Early approaches to bibliometrics.Journal of the American Society for Information Science, 38(2), 127–129.

    Google Scholar 

  • L. D. BROWN & J. C. GARDNER (1985). Using citation analysis to assess the impact of journals and articles on contemporary accounting research.Journal of Accounting Research, 23(1), 84–109.

    Google Scholar 

  • R. S. BURT (1978). Stratification and prestige among elite experts in methodological and mathematical sociology circa 1975.Social Networks, 1, 105–158.

    Google Scholar 

  • R. S. BURT & P. DOREIAN (1982). Testing a structural model of perception: Conformity and deviance with respect to journal norms in elite sociological methodology.Quality and Quantity, 16, 109–150.

    Google Scholar 

  • A. R. BUSS & J. R. McDERMOTT (1976). Ratings of psychology journals compared to objective measures of journal impact.American Psychologist, 31(9), 675–678.

    Google Scholar 

  • M. CARPENTER & F. NARIN (1973). Clustering of scientific journals.Journal of the American Society for Information Science, 23, 425–436.

    Google Scholar 

  • H. CASON & M. LUBOTSKY (1936). The influence and dependence of psychological journals on each other.Psychological Bulletin, 33(2), 95–103.

    Google Scholar 

  • A. CAWKELL (1978). Evaluating scientific journals with Journal Citation Reports: A case study in acoustics.Journal of American Society for Information Science, 29, 41–46.

    Google Scholar 

  • D. CHUBIN (1983).Sociology of sciences: An annotated bibliography on invisible colleges, 1972–1981, NY: Garland Publishing.

    Google Scholar 

  • J. COLE & S. COLE (1971). Measuring the quality of sociological research: Problems in the use of the Science citation index.American Sociologist, 6, 23–29.

    Google Scholar 

  • D. CRANE (1972).Invisible colleges: Diffusion of knowledge in scientific communities. Chicago: University of Chicago Press.

    Google Scholar 

  • D. CRANE (1974). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals.The American Sociologist, 2, 195–201.

    Google Scholar 

  • L. CUMMINGS & P. FROST (Eds), (1985).Publishing in the organizational sciences. Homewood, ILL: Irwin.

    Google Scholar 

  • J. DICKMAN (1984). How do secondary services select journals?Cleveland Clinic Quarterly, 51, 575–578.

    Google Scholar 

  • J. DICKMAN & G. PLATEAU (1983). Foreign literature challenges to Chemical Abstracts Service.Journal of Chemical Information and Computer Science, 23(2), 70–73.

    Google Scholar 

  • P. DOREIAN (1985). Structural equivalence in a psychology journal network.Journal of the American Society for Information Science, November, 411–417.

  • P. DOREIAN (1985). A measure of standing of journals in stratified networks.Journal of the American Society for Information Science, 8(5/6), 341–363.

    Google Scholar 

  • P. DOREIAN & T. FARARO (1985). Structural equivalence in a journal network.Journal of the American Society for Information Science, 36(1), 28–37.

    Google Scholar 

  • D. EDGE (1979). Quantitative measures of communication in science: A critical review.History of Science, 17, 102–133.

    Google Scholar 

  • E. GARFIELD (1979).Citation indexing: Its theory and application in science, technology and humanities. NY: Wiley.

    Google Scholar 

  • E. GARFIELD (1984). The 100 most-cited papers ever and how we select citation classics.Current Contents, 23, 3–9.

    Google Scholar 

  • W. GARVEY (1979).Communication: The essence of science. Facilitating information exchange among librarians, scientists, engineers, and students. NY: Pergamon Press.

    Google Scholar 

  • J. GOODRICH & C. ROLAND (1977). Accuracy of published medical reference citations.Journal of Technical Writing and Communication, 7, 15–19.

    Google Scholar 

  • M. GORDON (1982). Citation ranking versus subjective evaluation in the determination of journal hierarchies in the social sciences.Journal of the American Society for Information Science, 33, 55–57.

    Google Scholar 

  • W. HAGSTROM (1965).The scientific community. NY: Basic Books.

    Google Scholar 

  • P. J. HART (1987).The use and role of commercial online databases in four organizations. Unpublished PhD dissertation. Los Angeles: University of Southern California, Annenberg School of Communications.

  • G. HIRST (1978). Discipline impact factors: A method for determining core journal lists.Journal of the American Society for Information Science, 29, 171–172.

    Google Scholar 

  • R. HJERPPE (1980).A bibliography of bibliometrics and citationindexing and analysis. (TRITA-LIB-2013). Stockholm: Royal Institute of Technology Library.

    Google Scholar 

  • Institute for Scientific Information. (1985).Journal Citation Reports. Philadelphia: Institute for Scientific Information.

    Google Scholar 

  • D. LINDSEY (1978).The scientific publication system in social science. San Francisco: Josey-Bass.

    Google Scholar 

  • M. LINE (1985). Changes in rank lists of serials over time: Interluding versus citation data.College & Research Libraries, 46(1), 77–79.

    Google Scholar 

  • S. LOCK (1986).A difficult balance: Editorial peer review in medicine. Philadelphia, PA: Institute for Scientific Information.

    Google Scholar 

  • T. LUCE & D. JOHNSON (1978). Rating of education and psychological journals.Educational Researcher, 7, 8–10.

    Google Scholar 

  • K. MACE & H. WARNER (1973). Ratings of psychology journals.American Psychologist, 28(2), 184–186.

    Google Scholar 

  • P. McALLISTER; R. ANDERSON & F. NARIN (1980). Comparison of peer and citation assessment of the influence of scientific journals.Journal of the American Society for Information Science, 31, 147–152.

    Google Scholar 

  • N. MIDORIKAWA (1983). Citation analysis of physics journals: Comparison of subfield of physics.Scientometrics, 5(6), 361–374.

    Google Scholar 

  • F. NARIN; M. CARPENTER & N. BERLT (1972). Inter-relationships of scientific journals.Journal of the American Society for Information Science, 23, 323–331.

    Google Scholar 

  • F. NARIN & D. GARSIDE (1972). Journal relationships in special education.Exceptional Children, 38(9), 695–703.

    Google Scholar 

  • W. NORD (1985). Looking at ourselves as we look at others: An exploration of the publication system for organizational research. In: L. CUMMINGS & P. FROST (Eds),Publishing in the organizational sciences. Homewood, ILL: Irwin.

    Google Scholar 

  • W. PAISLEY (1984). Communication in the communication sciences. In: B. DERVIN & M. VOIGT (Eds),Progress in the communication sciences, Vol. 5 (pp. 1–43.) Norwood, NJ: Ablex.

    Google Scholar 

  • D. PETERS & S. CECI (1982). Peer review practices of psychological journals: The fate of published articles, submitted again.Behavioral and Brain Science, 2(5), 187–195. (Cited inLock, 1985).

    Google Scholar 

  • A. PORTER (1977). Citation analysis: Queries and caveats.Social Studies of Science, 7, 257–267.

    Google Scholar 

  • W. POWELL (1985).Getting into print: The decision-making process in scholarly publishing. Chicago: University of Chicago Press.

    Google Scholar 

  • B. REEVES & C. L. BORGMAN (1983). A bibliometric evaluation of core journals in communication research.Human Communication Research, 10(1), 119–136.

    Google Scholar 

  • R. E. RICE; C. L. BORGMAN & B. REEVES (1988). Citation networks of communication journals, 1977–1985.Human Communicatiin Research, 15(2), 256–283.

    Google Scholar 

  • J. RUSHTON (1984). The validity of citation counts.Bulletin of the British Psychological Society, 37, 33–36.

    Google Scholar 

  • W. SATARIANO (1978). Journal use in sociology: Citation analysis versus readership patterns.Library Quarterly, 48(3), 293–300.

    Google Scholar 

  • A. SINGLETON (1976). Journal ranking and selection: A review in physics.Journal of Documentation, 32(4), 258–289.

    Google Scholar 

  • W. SCOTT (1974). Interreferee agreement on some characteristics of manuscripts submitted to theJournal of Personality and Social Psychology.American Psychologist, 29, 698–702. (Cited in Bohannon, 1986).

    Google Scholar 

  • H. SMALL (1981). The relationship of information science to the social sciences: A co-citation analysis.Information Processing and Management, 17, 39–50.

    Google Scholar 

  • J. SMART (1983). Perceived quality and citation rates of education journals.Research in Higher Education, 19(2), 175–182.

    Google Scholar 

  • L. SMITH (1981). Citation analysis.Library Trends, Summer, 83–106.

  • N. SMITH & D. CAULLEY (1981). The evaluation of education journals through the study of citations.Educational Researcher, 10, 11–12.

    Google Scholar 

  • C. SO (1988). Citation networks of communication journals.Human Communication Research, 15(2), 236–255.

    Google Scholar 

  • SSCI Journal citation reports; a bibliometric analysis of social sciences journals in the ISI data base. Social Sciences Citation Index 1981 annual, vol. 7. (1981, 1984, 1985, 1986). Philadelphia: Institute for Scientific Information.

  • F. THORNE (1977). The Citation Index: Another case of spurious validity.Journal of Clinical Psychology, October, 33, 1157–1161.

    Google Scholar 

  • Ulrich's International Periodicals Guide; A classified Guide to a Selected List of Current Periodicals, Foreign and Domestic. 25th ed, 1986–87. (1986). NY: R. R. Bowker.

    Google Scholar 

  • H. WHITE & B. GRIFFITH (1982). Authors as markers of intellectual space: Co-citation on studies of science, technology and society.Journal of Documentation, 38(4), 255–272.

    Google Scholar 

  • M. WHITE & K. WHITE (1977). Citation analysis of psychology journals.American Psychologist, 32, 301–305.

    Google Scholar 

  • N. WHITMAN & B. EYRE (1985). The pattern of publishing previously rejected articles in selected journals.Family Medicine, 17(1), 26–28.

    Google Scholar 

  • S. Jr. WIBERLEY (1982). Journal rankings from citation studies: A comparison of national and local data from social work.Library Quarterly, 52(4), 348–359.

    Google Scholar 

  • M. E. WILLIAMS (Ed.), (1985).Computer-Readable Databases: A Directory and Sourcebook, Vols 1–2. Chicago: American Library Association.

    Google Scholar 

  • L. ZHIGNESSE & C. OSGOOD (1967). Bibliographic citation characteristics of the psychological journal network in 1950 and 1960.American Psychologist, 22, 778–791.

    Google Scholar 

  • J. ZIMAN (1968).Public knowledge. Cambridge, England: Cambridge University Press.

    Google Scholar 

  • H. ZUCKERMAN & R. MERTON (1971). Patterns of evaluation in science: Institutionalization, structure, and functions of the referee system.Minerva, 9, 66–100. (Cited in BOHANNON 1986.)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rice, R.E., Borgman, C.L., Bednarski, D. et al. Journal-to-journal citation data: Issues of validity and reliability. Scientometrics 15, 257–282 (1989). https://doi.org/10.1007/BF02017202

Download citation

  • Received:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02017202

Keywords

Navigation