Research on using ANP to establish a performance assessment model for business intelligence systems

https://doi.org/10.1016/j.eswa.2008.03.004Get rights and content

Abstract

In order to compete in the rigorous environment, the electronization has enabled business to deploy business intelligence (BI) systems for the purpose of decision-making. However, to avoid the ineffective experiences during the deployment, it is important to clarify the impact factors of a BI system and find out a suitable assessment method to evaluate the performance of BI systems. In this paper, an analytic network process (ANP) based assessment model was constructed to assess the effectiveness of BI systems. Furthermore, an expert questionnaire was used to filter out useful performance matrices, used as the sub-criteria of the ANP model. Finally, a real case was analyzed using the constructed ANP-based effectiveness assessment model for Business Intelligence systems. The results indicate that the most critical factors that impact the effectiveness of a BI system are: output information accuracy, conformity to the requirements, and support of organizational efficiency. Utilizing this model to assess the BI performance of the studied case, it reveals that 24% improvement in effectiveness has been reached, which consists with the perception of the management level. Therefore, this effectiveness assessment model can be used to evaluate the performances of a BI system. It can also provide performance indices and improvement directions for BI users and vendors, respectively, for the total succession in system effectiveness and satisfaction.

Introduction

Traditional enterprises may normally face issues such as the overflow of data, the lack of information, the lack of knowledge and insufficiency of reports. Therefore, in order to make prompt decision within the shortest period of time possible to keep pace with the situation, high levels of management commonly make decisions based on their experiences, leading to the ever-increasing risk of decision failure while lowering the value of the decision itself. As worldwide competition is maturating, past decision-making modes can no longer satisfy the requirements of enterprises for decision efficiency and benefits; enterprises must make good use of electronic tools to quickly extract useful information from huge volume of data by providing the skills of fast decision-making (Rakar & Jovan, 2004). The way to promote the electronization solutions from the operational level to the decision making level is a topic enterprises cannot avoid in the face of the next wave of electronization. The information system applied within the enterprises should be able to demonstrate the data or information with accuracy and in real-time, in order to expedite the speed of processing and decision-making. Existing electronization software package can provide a set of complete solutions for the operation and management processes of enterprises. However, the effects of the implementation of electronization tools vary that the probability of failure is higher than that of the success (Ward, Hemingway, & Daniel, 2005). Therefore, defining the performance of information tool and laying down related assessment criteria is an important issue that has to be tackled for the deployment of electronization.

Business intelligence (BI) is the tool used by enterprises to collect, manage and analyze structural and non-structural data and information by taking advantage of modern information technology (IT). It utilizes a substantial amount of collected data during the daily operational processes, and transforms the data into information and knowledge to avoid the supposition and ignorance of the enterprises (Wang, 2005). Under the speed-oriented operation mode, in order to improve management effects and performance, BI will surely become the tool enterprises would like to actively deploy as well as the solution that can bring enterprises competitive edge. However, current BI application is still at its fledging stage and most of the enterprises fall short of sufficient understanding towards BI (Wang, 2005); currently, research on conducting performance evaluation for the implementation of BI system is scarce, not to mention the analysis of on-line performance. Beside that, managers usually have to measure all the pros and cons to achieve a balance in assessing the performances of BI/IT systems. Different end users and IT people adopt different performance measurement criteria. Therefore, it is a significant issue to implement across-the-board considerations to incorporate different viewpoints and perspectives from manifold experts in BI development and usage into the choice for assessing BI performance effectiveness.

In order to lower the failure risk after implementation, it is necessary to conduct in-depth discussion for the aforementioned issue. Therefore, this research starts by analyzing BI benefits, takes advantage of analytic network process (ANP) to discuss BI effectiveness and related performance assessment indications. The results thus provide enterprises that are interested in deploying BI systems with a consistent and effective assessment model for future BI implementation while serving as a direction of future improvement and enhancement for BI software suppliers and consulting companies.

The remainder of this paper is organized as follows. Section 2 presents the related studies regarding to this research. Then, the research theory and method is presented in Section 3. Section 4 demonstrates the proposed architecture for assessing the performance of BI systems. An empirical research and related analysis is illustrated in Section 5. Then a case study on a global supplier of computer peripherals is described in Section 6, and the summary and conclusions are drawn finally.

Section snippets

Information systems

With the demands for information technology, application software and enterprise information tactics constantly are enhanced and expanded. The deployment of SCM, ERP, CRM systems, etc. has become mature, and the growth of business intelligence information system will become a new direction for enterprises’ electronization construction (Chung, Lee, & Pearn, 2005).

Research theory and method

This research method is consisted of three parts. The first part chooses the factors that influence the BI system’s effects as the foundation of the ANP analysis method. In this section, we sort out the key elements that influence the effects of information systems after gathering kinds of documents, and resort to the experiences and opinions acquired from questionnaire for experts. The second part adopts the selected elements to build up the ANP model. The third part is for case study. Based

Selection of impact factors

The ANP assessment model is aimed to assist enterprises to evaluate the effectiveness of a BI system. It provides the effective check and the effective analysis to those enterprises that have implemented or are going to implement a BI system, and improves the usability and satisfaction of BI system. In this research, the key elements that influence the performances of an information system are constructed after interviewing those seniors in the information department of industrial sectors and

Expert questionnaire

This paper utilizes ANP model to design expert questionnaire, and appraises the relative importance of criteria as numbers using the fundamental scale of the AHP, shown in Table 2 (Saaty, 2005).

In the questionnaire distribution phase, to avoid any ambiguity or hard readability that may affect the external nature of answers given by the interviewees, and to facilitate the understanding about this ANP model, our researchers physically visited the interviewees to make an on-site survey. The

Case study

To verify the effectiveness of the ANP model, we studied a company that has implemented the BI system as an example. Through the interview, the process of implementing the BI system and using information can be presented.

Conclusions

In recent years, all enterprises look for an efficient and effective information system as the tool to obtain competitive advantage. To lower operational costs and retain competitiveness, many enterprises expect to implement the BI system, integrate the internal and external data of the enterprises, interpret the data, and transfer them into useful information. However, the implementation of information system can not make distinctive effectiveness without suitable evaluation indicators. Thus,

References (23)

  • B. Ives et al.

    The measurement of user information satisfaction

    Communications of the ACM

    (1983)
  • Cited by (95)

    • Analytic network process: An overview of applications

      2020, Applied Mathematics and Computation
    • Evaluation of watershed health using Fuzzy-ANP approach considering geo-environmental and topo-hydrological criteria

      2019, Journal of Environmental Management
      Citation Excerpt :

      The weights of criteria were determined using the analytical network process (ANP) method in the SuperDecisions software. Studies have shown that the ANP is the most appropriate approach among multi-criteria methods (e.g., technique for ordering performance by similarity to ideal solution (TOPSIS) and analytical hierarchy process (AHP)), because it considers the inter-relationship between criteria and also sub-criteria (Lin et al., 2009; Kucukaltan et al., 2016). Recent studies have indicated that there are direct and/or indirect relationships between factors considered in this study, for example, the effect of climate on water quality, local morphometry, and erosion or other parameters (Chorley, 1957; Line et al., 2002; Dalzell et al., 2004; Zektser et al., 2007; Olson, 2012).

    • Perceptions regarding active management of the Cross-timbers forest resources of Oklahoma, Texas, and Kansas: A SWOT-ANP analysis

      2019, Land Use Policy
      Citation Excerpt :

      Accordingly, SWOT-ANP is the better approach for this study. Although the ANP is frequently used for business management applications (Feglar et al., 2006; Lin et al., 2009; Mu, 2006), it is still an emerging methodology in natural resource management and only a handful of the studies have used it. For example, Catron et al. (2013) used SWOT-ANP to examine the forest-based bioenergy industry in Kentucky, USA while Dağdeviren and Eraslan (2008) utilized this model to determine strategic energy policies in Turkey.

    View all citing articles on Scopus
    View full text