Skip to main content
SpringerLink
Log in

Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Journal rankings and journal ratings are important to governments, research institutes, and scientific research in general, and they frequently serve as the criteria for evaluating research performance to determine whether specific researchers will receive promotions and/or earn research grants. However, the only widely adopted journal assessment method is known as impact factor (IF), which focuses on citations in academic journals. However, IF disregards the technological applications and value of academic journals. In this article, we propose a method to rank academic journals that utilizes non-patent references in patent documents. We also compare the differences between journal rankings derived by using IF with those derived from the Intellectual Property Citation Index (IPCI) across different fields; moreover, some fields contain positive and significant correlations between IF and the IPCI. The results of this study offer a new perspective from which to assess the technological value of academic journals, particularly those in the technological and scientific fields. This study considers linkages among science and technology and the needs of the stakeholders in journal assessment to shed light on journal assessment and journal ranking methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abramo, G., Andrea D’Angelo, C., & Di Costa, F. (2009). Mapping excellence in national research systems: The case of Italy. Evaluation Review, 33(2), 159–188. doi:10.1177/0193841X08322871.

    Article  Google Scholar 

  • Agrawal, V. K. (2002). Constituencies of journals in production and operations management: Implications on reach and quality. Production and Operations Management, 11(2), 101–108. doi:10.1111/j.1937-5956.2002.tb00485.x.

    Article  Google Scholar 

  • Althouse, B. M., West, J. D., Bergstrom, C. T., & Bergstrom, T. (2009). Differences in impact factor across fields and over time. Journal of the American Society for Information Science and Technology, 60(1), 27–34. doi:10.1002/asi.20936.

    Article  Google Scholar 

  • Baum, J. A. C. (2011). Free-riding on power laws: Questioning the validity of the Impact Factor as a measure of research quality in organization studies. Organization, 18(4), 449–466. doi:10.1177/1350508411403531.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. doi:10.1108/00220410810844150.

    Article  Google Scholar 

  • Brusoni, S., Criscuolo, P., & Geuna, A. (2005). The knowledge bases of the world’s largest pharmaceutical groups: What do patent citations to non-patent literature reveal? Economics of Innovation and New Technology, 14(5), 395–415. doi:10.1080/1043859042000307356.

    Article  Google Scholar 

  • Buela-Casal, G. (2004). Assessing the quality of articles and scientific journals: Proposal for weighted impact factor and a quality index. Psychology in Spain, 8(1), 60–76.

    Google Scholar 

  • Callaert, J., Grouwels, J., & Looy, B. (2012). Delineating the scientific footprint in technology: Identifying scientific publications within non-patent references. Scientometrics, 91(2), 383–398. doi:10.1007/s11192-011-0573-9.

    Article  Google Scholar 

  • Callaert, J., Van Looy, B., Verbeek, A., Debackere, K., & Thijs, B. (2006). Traces of prior art: An analysis of non-patent references found in patent documents. Scientometrics, 69(1), 3–20. doi:10.1007/s11192-006-0135-8.

    Article  Google Scholar 

  • Callaham, M., Wears, R. L., & Weber, M. E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. The Journal of the American Medical Association, 287(21), 1733–1818. doi:10.1001/jama.287.21.2847.

    Google Scholar 

  • Campanario, J. M. (2011). Large increases and decreases in journal impact factors in only 1 year: The effect of journal self-citations. Journal of the American Society for Information Science and Technology, 62(2), 230–235. doi:10.1002/asi.21457.

    Article  Google Scholar 

  • Catling, J. C., Mason, V. L., & Upton, D. (2009). Quality is in the eye of the beholder? An evaluation of impact factors and perception of journal prestige in the UK. Scientometrics, 81(2), 333–345. doi:10.1007/s11192-009-2124-1.

    Article  Google Scholar 

  • Chang, C. M., & Hung, S. Y. (2004). Distinct journal preference of successful EC researchers: A citation analysis. Electronic Commerce Studies, 2(1), 1–18.

    MathSciNet  Google Scholar 

  • Chapron, G., & Husté, A. (2006). Open, fair, and free journal ranking for researchers. BioScience, 56(7), 558–559. doi:10.1641/0006-3568(2006)56[558:OFAFJR]2.0.CO;2.

  • Christensen, H. F., Ingwersen, P., & Wormell, I. (1997). Online determination of the journal impact factor and its international properties. Scientometrics, 40(3), 529–540. doi:10.1007/BF02459298.

    Article  Google Scholar 

  • Claro, J., & Costa, C. A. V. (2010). A made-to-measure indicator for cross-disciplinary bibliometric ranking of researchers performance. Scientometrics, 86(1), 113–123. doi:10.1007/s11192-010-0241-5.

    Article  Google Scholar 

  • Cohen, B. (2007). Journal ratings and footprints: a North American perspective of organizations and the natural environment journal quality. Business Strategy and Environment, 16(1), 64–74. doi:10.1002/bse.461.

    Article  Google Scholar 

  • Coleman, A. (2007). Assessing the value of a journal beyond the impact factor. Journal of the American Society for Information Science and Technology, 58(8), 1148–1161. doi:10.1002/asi.20599.

    Article  Google Scholar 

  • Crewe, I., & Norris, P. (1991). British and American journal evaluation: Divergence or convergence? Political Science and Politics, 24(3), 524–531.

    Article  Google Scholar 

  • Crews, J. M., McLeod, A., & Simkin, M. G. (2009). Journal self-citation XII: The ethics of forced journal citations. Communications of AIS, 25(12), 97–110.

    Google Scholar 

  • Epstein, W. M. (2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice, 14(6), 450–458. doi:10.1177/1049731504265838.

    Article  Google Scholar 

  • Extejt, M. M., & Smith, J. E. (1990). The behavioral sciences and management: An evaluation of relevant journals. Journal of Management, 15(3), 539–551. doi:10.1177/014920639001600302.

    Article  Google Scholar 

  • Gao, X., & Guan, J. (2009). Networks of scientific journals: An exploration of Chinese patent data. Scientometrics, 80(1), 283–302. doi:10.1007/s11192-007-2013-4.

    Article  Google Scholar 

  • García, J. A., Rodriguez-Sánchez, R., & Fdez-Valdivia, J. (2011). Overall prestige of journals with ranking score above a given threshold. Scientometrics, 89(1), 229–243. doi:10.1007/s11192-011-0442-6.

    Article  Google Scholar 

  • Geary, J., Marriott, L., & Rowlinson, M. (2004). Journal rankings in business and management and the 2001 research assessment exercise in the UK. British Journal of Management, 15(2), 95–141. doi:10.1111/j.1467-8551.2004.00410.x.

    Article  Google Scholar 

  • Gordon, M. D. (1982). Citation ranking versus subjective evaluation in the determination of journal hierachies in the social sciences. Journal of the American Society for Information Science, 33(1), 55–57. doi:10.1002/asi.4630330109.

    Article  Google Scholar 

  • Haddow, G., & Genoni, P. (2010). Citation analysis and peer ranking of Australian social science journals. Scientometrics, 85(2), 471–487. doi:10.1007/s11192-010-0198-4.

    Article  Google Scholar 

  • Haynes, J. P. (1983). An empirical method for determining core psychology journals. American Psychologist, 38(8), 959–961. doi:10.1037/0003-066X.38.8.959.

    Article  Google Scholar 

  • Hicks, D., & Wang, J. (2013). The New York times as a resource for mode 2. Science, Technology and Human Values, 38(6), 851–877. doi:10.1177/0162243913497806.

    Article  Google Scholar 

  • Huang, M.-H., Huang, W.-T., & Chen, D.-Z. (2014). Technological impact factor: An indicator to measure the impact of academic publications on practical innovation. Journal of Informetrics, 8(1), 241–251. doi:10.1016/j.joi.2013.12.004.

    Article  Google Scholar 

  • Jones, M. J., Brinn, T., & Pendlebury, M. (1996). Journal evaluation methodologies: a Balanced response. Omega, 24(5), 607–612. doi:10.1016/0305-0483(96)00038-2.

    Article  Google Scholar 

  • Kao, C., Lin, H.-W., Chung, S.-L., Tsai, W.-C., Chiou, J.-S., Chen, Y.-L., et al. (2008). Ranking Taiwanese management journals: A case study. Scientometrics, 76(1), 95–115. doi:10.1007/s11192-007-1895-5.

    Article  Google Scholar 

  • Kothari, T., & Lahiri, S. (2012). Yesterday, today and tomorrow: An overview of research publications in the Journal of International Management. Journal of International Management, 18(1), 102–110. doi:10.1016/j.intman.2011.09.001.

    Article  Google Scholar 

  • Leydesdorff, L., & Bornmann, L. (2011). Integrated impact indicators compared with impact factors: An alternative research design with policy implications. Journal of the American Society for Information Science and Technology, 62(11), 2133–2146. doi:10.1002/asi.21609.

    Article  Google Scholar 

  • Lo, S.-C. S. (2009). Scientific linkage of science research and technology development: A case of genetic engineering research. Scientometrics, 82(1), 109–120. doi:10.1007/s11192-009-0036-8.

    Article  Google Scholar 

  • Marx, W., Bornmann, L., & Cardona, M. (2010). Reference standards and reference multipliers for the comparison of the citation impact of papers published in different time periods. Journal of the American Society for Information Science and Technology, 61(10), 2061–2069. doi:10.1002/asi.21377.

    Article  Google Scholar 

  • McCain, K. W. (1991). Core journal networks and cocitation maps: New bibliometric tools for serials research and management. The Library Quarterly, 61(3), 311–336.

    Article  Google Scholar 

  • McMillan, G. S., Narin, F., & Deeds, D. L. (2000). An analysis of the critical role of public science in innovation: The case of biotechnology. Research Policy, 29(1), 1–8. doi:10.1016/S0048-7333(99)00030-X.

    Article  Google Scholar 

  • Meyer, M. (2002). Tracing knowledge flows in innovation systems. Scientometrics, 54(2), 193–212. doi:10.1023/A:1016057727209.

    Article  Google Scholar 

  • Michel, J., & Bettels, B. (2001). Patent citation analysis: A closer look at the basic input data from patent search reports. Scientometrics, 51(1), 185–201.

    Article  Google Scholar 

  • Moed, H., Colledge, L., Reedijk, J., Moya-Anegon, F., Guerrero-Bote, V., Plume, A., et al. (2012). Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way. Scientometrics, 92(2), 367–376. doi:10.1007/s11192-012-0679-8.

    Article  Google Scholar 

  • Narin, F., & Olivastro, D. (1992). Status report: Linkage between technology and science. Research Policy, 21(3), 237–249. doi:10.1016/0048-7333(92)90018-Y.

  • Narin, F., Hamilton, K. S., & Olivastro, D. (1997). The increasing linkage between U. S. technology and public science. Research Policy, 26(3), 317–330. doi:10.1016/S0048-7333(97)00013-9.

    Article  Google Scholar 

  • Narin, F., & Noma, E. (1985). Is technology becoming science? Scientometrics, 7(3–6), 369–381. doi:10.1007/BF02017155.

    Article  Google Scholar 

  • Palacios-Huerta, I., & Volij, O. (2004). The measurement of intellectual influence. Econometrica, 72(3), 963–977. doi:10.1111/j.1468-0262.2004.00519.x.

    Article  MATH  Google Scholar 

  • Pichappan, P. (1995). A dual refinement of journal self-citation measures. Scientometrics, 33(1), 13–21. doi:10.1007/BF02020772.

    Article  Google Scholar 

  • Podsakoff, P. M., Mackenzie, S. B., Bachrach, D. G., & Podsakoff, N. P. (2005). The influence of management journals in the 1980s and 1990s. Strategic Management Journal, 26(5), 473–488. doi:10.1002/Smj.454.

    Article  Google Scholar 

  • Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation, 19(5), 347–360. doi:10.3152/095820210x12809191250889.

    Article  Google Scholar 

  • Robinson, L. M., & Adler, R. (1981). Measuring the impact of marketing scholars and institutions: An analysis of citation frequency. Journal of the Academy of Marketing Science, 9(1), 147–162. doi:10.1007/BF02723573.

    Article  Google Scholar 

  • Rousseau, R. (2002). Journal evaluation: Technical and practical issues. Library Trends, 50(3), 418–439.

    Google Scholar 

  • Schoenmakers, W., & Duysters, G. (2010). The technological origins of radical inventions. Research Policy, 39(8), 1051–1059. doi:10.1016/j.respol.2010.05.013.

    Article  Google Scholar 

  • Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 497–513.

    Article  Google Scholar 

  • Shearer, B. A., Lundeberg, M. A., & Coballes-Vega, C. (1997). Making the connection between research and reality: Strategies Teachers use to read and evaluate journal articles. Journal of Educational Psychology, 89(4), 592–598. doi:10.1037/h0092690.

    Article  Google Scholar 

  • Singleton, A. (1976). Journal ranking and selection: A review in physics. Journal of Documentation, 32(4), 258–289. doi:10.1108/eb026628.

    Article  Google Scholar 

  • Sombatsompop, N., & Markpin, T. (2005). Making an equality of ISI impact factors for different subject fields. Journal of the American Society for Information Science and Technology, 56(7), 676–683. doi:10.1002/asi.20150.

    Article  Google Scholar 

  • Sutter, M., & Kocher, M. G. (2001). Tools for evaluating research output: Are citation-based rankings of economics journals stable? Evaluation Review, 25(5), 555–566. doi:10.1177/0193841x0102500503.

    Article  Google Scholar 

  • Tahai, A., & Meyer, M. J. (1999). A revealed preference study of management journals’ direct influences. Strategic Management Journal, 20(3), 279–296.

    Article  Google Scholar 

  • Tijssen, R. J. W., Visser, M. S., & Van Leeuwen, T. N. (2002). Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3), 381–397. doi:10.1023/A:1016082432660.

    Article  Google Scholar 

  • Todorov, R., & Glanzel, W. (1988). Journal citation measures: A concise review. Journal of Information Science, 14(1), 47–56. doi:10.1177/016555158801400106.

    Article  Google Scholar 

  • Turban, E., Zhou, D., & Ma, J. (2004). A group decision support approach to evaluating journals. Information & Management, 42(1), 31–44. doi:10.1016/j.im.2003.12.003.

    Article  Google Scholar 

  • Tussen, R. J., Buter, R., & Van Leeuwen, T. N. (2000). Technological relevance of science: An assessment of citation linkages between patents and research papers. Scientometrics, 47(2), 389–412. doi:10.1023/A:1005603513439.

    Article  Google Scholar 

  • Ugaz, A. G. (2011). Drilling deeper into the core: an analysis of journal evaluation methodologies used to create the “Basic List of Veterinary Medical Serials,” third edition. Journal of Medical Library Association, 99(2), 145–152. doi:10.3163/1536-5050.99.2.006.

    Article  Google Scholar 

  • Vastag, G., & Montabon, F. (2002). Journal characteristics, rankings and social acculturation in operations management. Omega, 30(2), 109–126. doi:10.1016/S0305-0483(01)00061-5.

    Article  Google Scholar 

  • Verbeek, A., Debackere, K., Luwel, M., Andries, P., Zimmermann, E., & Deleus, F. (2002). Linking science to technology: Using bibliographic references in patents to build linkage schemes. Scientometrics, 54(3), 399–420. doi:10.1023/A:1016034516731.

    Article  Google Scholar 

  • Vinkler, P. (2002). Subfield problems in applying the Garfield (Impact) Factors in practice. Scientometrics, 53(2), 267–279. doi:10.1023/A:1014860726532.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Business and Management, Ming Chi University of Technology, No. 84, Gongzhuan Rd., Taishan Dist., New Taipei City, 24301, Taiwan

    Yi-Ching Liaw

  2. National Applied Research Laboratories, Science & Technology Policy Research and Information Center, 14F No. 106, Sec. 2, Heping E. Road, Taipei, Taiwan

    Te-Yi Chan & Chin-Yuan Fan

  3. School of Business, Macau University of Science and Technology, Avenida Wai Long, Taipa, Macau

    Cheng-Hsin Chiang

Authors
  1. Yi-Ching Liaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Te-Yi Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chin-Yuan Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cheng-Hsin Chiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cheng-Hsin Chiang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liaw, YC., Chan, TY., Fan, CY. et al. Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis. Scientometrics 101, 17–37 (2014). https://doi.org/10.1007/s11192-014-1337-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-014-1337-0

Keywords

Search

Navigation