Statistical profiles of highly-rated learning objects

Abstract

The continuously growth of learning resources available in on-line repositories has raised the concern for the development of automated methods for quality assessment. The current existence of on-line evaluations in such repositories has opened the possibility of searching for statistical profiles of highly-rated resources that can be used as priori indicators of quality. In this paper, we analyzed 35 metrics in learning objects refereed inside the MERLOT repository and elaborated profiles for these resources regarding the different categories of disciplines and material types available. We found that some of the intrinsic metrics presented significant differences between highly rated and poorly-rated resources and that those differences are dependent on the category of discipline to which the resource belongs and on the type of the resource. Moreover, we found that different profiles should be identified according to the type of rating (peer-review or user) under evaluation. At last, we developed an initial model using linear discriminant analysis to evaluate the strength of relevant metrics when performing an automated quality classification task. The initial results of this work are promising and will be used as the foundations for the further development of an automated tool for contextualized quality assessment of learning objects inside repositories.

Introduction

Learning objects (LOs) are often defined as digital entities that can be used and reused in the process of learning and education, and are considered by many as the cornerstones for the widespread development and adoption of e-learning initiatives. Several initiatives and proposals for LO quality evaluation have been discussed in the last years (Díaz et al., 2002, Kay and Knaack, 2009, Nesbit et al., 2002, Nesbit et al., 2003, Vargo et al., 2003, Williams, 2000) nevertheless, there is still no consensus on what constitutes a good quality LO, neither which is the best way of conducting the process of evaluation. In part, this can be attributed to the heterogeneous and multi-faceted nature of these resources. As they can differ in several aspects (size, granularity, technology used, type, metadata standard, instructional design, duration, etc.) (Churchill, 2007), it is reasonable to assume that the quality criteria and the ways of measurement them will also differ accordingly to these many aspects. Moreover, the different evaluation approaches also reflect the many particular contexts of usage for the learning objects, as each one of them usually measures quality from the perspective of “a given repository, a country or a community of users” (Vuorikari, Manouselis, & Duval, 2008). In any case, the continuous growing of educational resources on the internet has turned impractical to rely only on human effort to classify good quality learning materials, and has raised the concern about the development of new automated techniques and tools that could be used to complement the existing approaches in order to relieve manual work. The actual abundance of resources inside repositories (Ochoa & Duval, 2009) and the availability of contextual evaluations in some of them have opened the possibility of seeking for intrinsic metrics of learning objects that could be used as indicators of quality. This means to say that learning objects could be “mined” and quantitative measures of good and not-good resources could be compared in order to discover intrinsic attributes associated with quality, thus allowing the creation of statistical profiles of good and poor resources that could serve as the basis for quality prediction. In fact, such approach was previously successfully applied to automatically analyze the usability of websites by Ivory and Hearst (2002b). It is known that learning object quality can be considered a more complex construct than usability as the latter is included in existing instruments as LORI (Nesbit et al., 2003) as one out of the several attributes considered, and that we cannot take for granted that the same correlations found by them are still applicable to ratings of learning objects (even though it may be hypothesized that the former affects the latter to some extent). So a first step in finding statistical profiles for highly-rated learning objects is exploring evidence on potential intrinsic measures which contribute to the classification of learning object quality, taking as a point of departure some of the ones that were identified for usability and others also found in related literature. This was initially done in the specific context of learning objects by García-Barriocanal & Sicilia (2009), where the authors preliminarily explored statistical profiles of highly-rated learning objects referenced on MERLOT repository¹. In that work, the authors contrasted four basic metrics (number of links, size in bytes, number of images and number of personal collections; the last one as a factor of contrast) against the main categories of disciplines available in MERLOT (Arts, Business, Education, Humanities, Mathematics and Statistics, Science and Technology, and Social Sciences) and have found initial (but still unclear) evidence that the number of images is normally associated with the ratings of a learning object, and could consequently be considered as a possible intrinsic measure that could be used to assess quality.

Even though automated analysis cannot replace traditional inspection techniques, it carries the potential of offering an inexpensive and time saving mechanism to a priori explore the quality of materials, therefore complementing other existing approaches. This paper aims to offer the very first foundations for the development of such tool by contrasting intrinsic metrics of highly-rated and poorly-rated learning objects stored in MERLOT and identifying which metrics are mostly associated with rated resources in the context of this repository. Such metrics can further serve as possible input variables to be used inside the tool. The deployment of such automated tool would certainly improve the general quality of the services provided by the repository regarding the processes of searching, selecting and recommending good quality materials. Contributors could, for instance, benefit of such new feature by evaluating beforehand the quality of their resources, which would allow their improvement through the use of the quality metrics referenced by the tool. We believe this would positively affect their intention to contribute to the repository with new resources. Moreover, it is known that many resources included by teachers inside their virtual courses are links to external websites (González-Videgaray, Hernández-Zamora, & del-Río-Martínez, 2009), such tool would allow educators to have a complementary perspective of quality of the resources before adding them into their courses.

In here, we extend the work of García-Barriocanal & Sicilia (2009) in a number of ways. First, a much larger number of metrics are used in the analysis. Second, in the previous work the metrics were computed only for the main page² of the resources, whereas in here we computed all internal websites up to a 2 level depth from the root node. Third, as MERLOT also classifies the materials according to their type (such as: Animation, Simulation, Drill, etc), and these different types normally present distinct features according to the literature (Churchill, 2007), we have also performed an analysis to contrast the metrics for the materials regarding this classification. And fourth, we initially tested the employment of some of these metrics in the composition of different Linear Discrimant models in order to verify their accuracy in the process of discriminating LOs regarding their quality.

The rest of this paper is structured as follows. Section 2 describes existing previous work regarding automated evaluation of LOs and some potential intrinsic measures that can be automatically extracted from the resources. Section 3 describes the data and the methodology applied for this study, as well as the statistical profiles encountered so far. Section 4 briefly explores the development of models to predict quality based on those profiles. Finally, section 5 presents the conclusions and limitations of this study, as well as some open possibilities for future work.