Elsevier

Information & Management

Volume 42, Issue 1, December 2004, Pages 31-44
Information & Management

A group decision support approach to evaluating journals

https://doi.org/10.1016/j.im.2003.12.003Get rights and content

Abstract

One of the most important decisions made in academic institutions, research organizations, and government agencies is the grading or ranking of journals for their academic values. Current methods for evaluating journals use either a subjective (e.g., experts’ judgments on journals) or objective approach (e.g., impact factors of journals), or an informal mix of the two. This paper presents a formal procedure that integrates objective and subjective judgments to provide a comprehensive method. The procedure is based on a fuzzy set approach that deals with the imprecise and missing information inherent in the evaluation process. The system was tested in Hong Kong in an assessment of faculty research productivity. Similar assessments exist in the UK, Singapore, and other countries. The proposed model can also be used for similar decisions that involve subjective and objective information.

Introduction

In a university setting, evaluating the quality of academic journals is required for several personnel decisions, such as recruiting, promotion, tenure, and retention. Such evaluation is also performed for merit increases and for allocation of research funding. Many institutions use formal grading or point systems for such evaluations. In many cases, the grading decision is made by a group (a committee or panel) which complicates the decision process, since a consensus is attempted. For years, researchers in several disciplines have attempted to find appropriate approaches for such evaluations. Unfortunately, there is no consensus on how best to conduct the evaluation. The methodologies proposed here (see Table 1) can be classified as either subjective or objective, depending on how the decision information is obtained and used.

The objective approaches are usually based on some form of citation counts over a certain time period. For example, Holsapple et al. [11] employed a citation analysis methodology to rank information systems research journals. This is publicly available in a form of “total cites”, “immediacy index”, “total articles”, “cited half-life” and “impact factor”. Citation information is published periodically in sources such as the Journal Citation Reports, a CD-ROM database available in many libraries.

The subjective approach, also called perception analysis approach, solicits information from experts such as academic staff, deans, or department heads (see [23]). The collected information is compiled and the average is used to rank the journals. Thus, the final ranking of the journals reflects the opinions of the group members.

However, evaluations and rankings of journals determined by subjective approaches can be influenced by biases [7] or by lack of sufficient knowledge or experience by some group members. Therefore, it is likely that neither the subjective, nor the objective approaches are the best method of evaluation. Thus, it makes sense to combine the two. However, despite considerable research conducted on evaluation, there is almost no research on how to integrate the two approaches. One exception is Forgionne and Kohli [8] who suggested measuring the quality of journals by 24 objective variables, giving subjective weights for each using the AHP software. Their methodology may not, however, be applicable to situations like ours due to the complexity of the model. In addition, there is very little work on two related issues: how to deal with incomplete subjective or objective information and how to transform the evaluations into a tangible journal grade.

Our paper attempts to fill this gap by proposing a group decision support system (GDSS) that deals with these research issues. The system is based on a methodology developed to fit the process of funding research in Hong Kong; however, it can be easily modified to cover other situations. The Hong Kong process involves a combination of subjective information solicited from experts who are organized in disciplinary panels and objective information in the form of an impact factor. A journal’s impact factor for a specific year is defined as “the number of citations to articles published in this journal in the previous 2-year period divided by the total number of articles published in this journal in the previous 2-year period” [26].

Section snippets

The proposed decision support process

Due to incomplete and uncertain objective information (e.g., citation information may not be available for some journals), as well as lack of sufficient knowledge, experts may find it difficult to express their preferences precisely. Fuzzy set theory [25], which is widely used in decision making (e.g., [14], [15], [17]), is used here as a tool for solving the problem of imprecise subjective judgments and incomplete objective information.

The proposed decision support process is composed of the

The GDSS structure

Based on this assessment process, a GDSS was defined; it consists of the four major components shown in Fig. 2.

  • (a)

    The database: The database contains two types of objective data: historical and current. The historical data include the journal ranking that existed prior to the current evaluation. In Hong Kong, this will be 3 years old or less, while in other countries it may be 1–5 years old. In addition, the database includes the impact factors of the relevant journals at the time the most recent

Application in Hong Kong

To enhance research activities in Hong Kong, the University Grant Committee (UGC), a central government funding agency, has conducted research assessment exercises (RAE) once every 3 years. Similar exercises are conducted in the UK (e.g., see [5]). For this exercise, local universities form cost centers according to disciplines suggested by the UGC. Academic staff members are then assigned to these centers. Each staff member submits up to five recently (i.e., in the past 3-year period)

Evaluation

The proposed system was demonstrated to potential users in Hong Kong. The essentials of the system were also presented at two academic conferences. We used a simple questionnaire to test the participants’ perception to the proposed system. The results of a pilot study using 13 participants who had experienced the use of the old method in Hong Kong and 13 academics who participated in a conference and attended our presentation about the system, indicated the following:

On a Likert scale of 7 (1:

Acknowledgements

This research was partly supported by the National Natural Science Foundation of China and Hong Kong Research Grant Council Joint Funding Scheme (Project No. 9050137), and the Competitive Earmarked Research Grant (CERG), Hong Kong SAR (Project Nos. 9040709 and 9040825) and Strategic Research Grant of City University of Hong Kong (Project Nos. 7100288 and 7001143). Many thanks to the anonymous reviewers for their contributions to improve this manuscript.

Efraim Turban (MBA, PhD, University of California at Berkeley) is Professor of Information Systems at City university of Hong Kong. Prior to that, he served on the faculty of several universities, including Lehigh University, Florida International University, and the University of Southern California. Prof. Turban is the author of over 100-refereed articles in journals such as Management Sciences, MIS Quarterly, Operations Research, Journal of MIS and IEEE Transaction on Engineering Management.

References (26)

  • J. Diaz et al.

    A note on the ranking of real estate research journals

    Real Estate Economics

    (1996)
  • M.M. Extejt et al.

    The behavioral sciences and management: an evaluation of relevant journals

    Journal of Management

    (1990)
  • L.Y. Flores et al.

    Trend analyses of major contributions in the counseling psychologist cited from 1986 to 1996: impact and implications

    Counseling Psychologist

    (1999)
  • Cited by (42)

    • A fuzzy recommender system based on the integration of subjective preferences and objective information

      2014, Applied Soft Computing Journal
      Citation Excerpt :

      This study proposes a novel CF system which integrates subjective and objective information to generate recommendations for an active consumer. The subjective information is comprised of opinions solicited from domain experts and opinion leaders [11,13,20,29]. The objective information clarifies the active user's preferences based on those of similar users and past experience.

    • A multicriteria decision analysis model for faculty evaluation

      2012, Omega
      Citation Excerpt :

      Subjective approaches, on the other hand, can be influenced by personal biases or by some lack of or insufficient knowledge or experience by some group members [17]. There has been little research on how to integrate objective and subjective approaches adequately [17]. Given that faculty evaluation implicitly incorporates many beliefs about academic careers and institutional policy, generates different costs and shapes the power relationships between stakeholders, as well as interacts with the balance between personal and departmental goals in academia [15], it is not an easy task to build and promote changes in evaluation systems [18].

    View all citing articles on Scopus

    Efraim Turban (MBA, PhD, University of California at Berkeley) is Professor of Information Systems at City university of Hong Kong. Prior to that, he served on the faculty of several universities, including Lehigh University, Florida International University, and the University of Southern California. Prof. Turban is the author of over 100-refereed articles in journals such as Management Sciences, MIS Quarterly, Operations Research, Journal of MIS and IEEE Transaction on Engineering Management. He also co-authored 20 books including Decision Support and Intelligent Systems, Electronic Commerce: A managerial perspective, and Information Technology for Management. Prof. Turban areas of research are quantitative decision support and implementation of Electronic commerce.

    Duanning Zhou is Assistant Professor at the Accounting & Information Systems Department, School of Business and Public Administration, Eastern Washington University, Washington. He received his PhD in Information Systems from City University of Hong Kong in 2000. His current research interests include group decision support systems (GDSS), fuzzy systems, system analysis & design, and electronic commerce. He has papers published in IEEE Transactions on Education, Information & Management, and Theoretical Computer Science.

    Jian Ma is Associate Professor in the Department of Information Systems at the City University of Hong Kong. He received his Doctor of Engineering degree in Computer Science from the Asian Institute of Technology in 1991. He was a Lecturer in the School of Computer Science and Engineering at the University of New South Wales, Australia, before joining the City University in 1993. Dr. Ma’s current research areas include electronic business systems, web-based decision support systems, object-oriented and component-based methods for information systems development. His past research has published in IEEE Transactions on Education, IEEE Transactions on Systems, Man and Cybernetics, Decision Support Systems and European Journal of Operational Research.

    View full text