Developing a new framework for evaluating e-learning systems: integrating BSC and FAHP

E-learning is one area of strategic application of information technology (IT) in the educational field. In the past, massive investments in e-learning systems have neither been necessarily effective nor created value for e-learners. As a result, e-learning systems’ evaluation has become critical. Although many researchers have studied e-learning’s performance evaluation, there is little research on e-learning assessment, which uses pedagogical principles and organizational issues along with information systems (ISs) assessment measures. Thus, the purpose of this study is to consolidate multi-dimensional aspects of e-learning evaluation by applying balanced scorecard (BSC) as a strategic performance measurement method. This study further seeks a solution for designing a systematic approach, in which three equally important considerations must be balanced: organization, pedagogy and technology.

This paper introduces a framework for assessing e-learning systems by using fuzzy analytic hierarchy process (FAHP) and BSC. Owing to the growing use of BSC in strategic management, this study applies its four perspectives as a base for performance evaluation. To identify measures of each perspective, a comprehensive literature review has been conducted. Modifying the list of measures is completed in two phases: first by using a focus group consisting of six e-learning experts, and second, by using a survey analysis conducted among 256 Iranian e-learners. FAHP is applied to weigh and determine the rank-order of the measures.

The results propose integrating pedagogical, organizational and technical measures based on the BSC framework for evaluating e-learning systems. In the proposed framework, effective BSC perspectives and their rank-orders and weights are presented as a guideline for assessing e-learning systems in Iran.

In e-learning initiatives, organizational issues must be considered along with technological and pedagogical factors simultaneously. The proposed framework can assist educational institutions and organizations to identify weaknesses and strengths of e-learning projects and guide them to select appropriate strategies for progress.

A review of the e-learning literature shows that there are few substantive theoretical accounts, which adequately integrate multiple dimensions of e-learning evaluation; yet, most researchers view e-learning evaluation as the most difficult part in the e-learning implementation process. This study proposes a new conceptual framework to evaluate e-learning initiatives systematically. The main contribution of this study is to develop a new systematic approach for e-learning systems’ evaluation that integrates two well-established managerial methodologies (BSC and FAHP), and considers pedagogical, organizational and technological aspects synchronously.