An Analysis on the Recommendation Engine of a Course Introduction Module

Abstract

This paper is about the previously proposed recommendation engine of a course introduction module. The objective of this paper is to introduce the structure and determine the effect of the module. In addition to presenting the structure of the module, this paper presents a statistical analysis of the effect of the module, and confirms its statistically significant effect.

1 Objective

Generally speaking, information acquisition needs differ from person to person, and a recommendation engine is used to meet such needs. This paper describes the recommendation engine of a course introduction module described in [1]. The objective of this paper is to introduce the structure and effect of the course introduction module.

The paper first presents the background of this research, including related work. Then, the structure of the course introduction module is described. Finally, an analysis on the effect of the course introduction module is presented.

2 Background and Significance

This section presents the background and significance of this research. It first reviews previous studies and then describes the structure of the course introduction module and its recommendation engine.

2.1 Previous Studies

The structure of the course introduction module is based on previous studies of e-portfolios. The course introduction module in this paper and [1] can also be defined as an e-portfolio. According to [2], surveys using questionnaires seldom help improve lecture designs. The results obtained by questionnaires in general are too uniform to determine a student’s satisfaction and the attainment of their goals. However, an e-portfolio allows us to check these results. An analysis based on e-portfolios reveals why students are satisfied with courses. It also shows how much students have attained their goals thanks to the course.

The course introduction module of this paper was developed based on previous studies. This module unites two other e-learning modules of the content management system (CMS) Xoops. One module is an online examination and drill module, and the other is an audience response system (ARS) module. Both modules have their own portfolio functions that students can use review the material they have learned. Both modules are implemented in CMS. Moreover, CMS can be used after the students have graduated. This feature allows the modules to collect data after the end of the course. Therefore, e-portfolio functions that can use such data can reveal methods of learning that are effective long term. Moreover, CMS unites the two e-portfolio functions. The course introduction module is also designed to reinforce this unity (Fig. 1).

Fig. 1.

CMS and e-portfolio functions

The course introduction module has a direct link with the e-portfolio pages of the two modules. Hence, the course introduction module can not only show course objectives and their attainment but also enables the students to review their learning precisely. This structure hence improves learning. In addition, this module has a recommendation engine that recommends proper courses. The function of a course introduction and e-portfolio was explained in [3]. The authors of [1] introduced the function of a recommendation engine for the course introduction module.

According to [3], a course introduction (e-portfolio) module must have the following characteristics.

• 1. The objectives that are checked by the course introduction module must be more abstract than those of the drill and ARS modules (concrete objectives are checked by the portfolio functions of the other two modules).

• 2. The module must allow students to set their own goals and level of effort.

• 3. The module must enable lecturers to set the objectives of their courses and the effort needed to attain them.

As [1] described, the recommendation engine can create recommendation data from the effort data and attainment data. This feature may help students formulate a learning plan.

2.2 Structure of the Module

Here, the structure of the module is described in depth. First, the main functions of the course introduction module are presented. Second, the calculation of the recommendation engine implemented in the module is shown.

The main functions of the module are shown in Fig. 2.

Fig. 2.

Course introduction module functions

As shown in Fig. 2, the administrators can set the master data of the courses and schedule. Each lecturer can then enter the detailed data for their courses. They can then set the objectives of their courses and the level of effort needed to attain them. Students can also set their objectives and level of effort. Hence students can compare their own objectives with the course objectives. This feature can help students create a learning plan.

After a course has ended, students can evaluate the course and check the attainment of their objectives. Lecturers can also evaluate the students and check the attainment of the objectives. Hence, students can compare their evaluation with those of the lecturers, i.e., students can analyze their learning more objectively.

As shown in [1], these data allow us to create recommendations using the following items.

• 1. Evaluation data of the course (5 degrees)

• 2. Credit data (2 degrees)

• 3. Effort required by students to obtain the objectives of the courses (10 degrees)

• 4. Attainments of goal (5 degrees)

The third item is based on the master data of the objectives, which is set by the administrators. Therefore, the relationship between the students’ objectives and courses’ objective is indirect. The other three items relate to the course and student data.

The data structure is shown in Fig. 3.

Fig. 3.

Data structure

Based on these items, some recommendation items are created in the same way as in the recommendation engine for the drill module described in [4]. The recommendations are provided by the course introduction module based on the following factors.

• 1. Similarity of the course evaluation

• 2. Credit data

• 3. Similarity of the objectives

• 4. Similarity of the attainments

The recommendation based on the similarity of course evaluation is calculated by Pearson’s correlation coefficient. In addition, those based on credit data are calculated by Tanimoto’s correlation coefficient. These methods are almost the same as those of the drill module [4].

However, the third and fourth recommendation factors are a bit different. Similarity scores are calculated beforehand. Pearson’s correlation coefficient between similarity scores is used to create the recommendation items. The similarity score for objectives is calculated as follows, which is the sum of the product of the student and course efforts:

$$ \mathop \sum \limits_{i = 1}^{n} CiSi $$

Here, C1 is the effort for the first objective of the course, and S1 is the effort for the first objective of the student.

The similarity score for attainments is created in almost the same way. It is the sum of the product of student and course attainments:

$$ \mathop \sum \limits_{i = 1}^{n} CiSi $$

Similarly, C1 is the first attainment of the course, and S1 is the first attainment of the student.

2.3 Use of the Module

This part describes how the module is used. An overview of its use is shown in Fig. 4.

Fig. 4.

Use of the course introduction module

First, the administrators must set the master data of the objectives and subjects. Then, they assign the subjects and schedule for the CMS group data.

After the master data of the objectives has been set, students can set their effort for the objectives (Fig. 5).

Fig. 5.

Screenshot of the student’s objective setting page

If the schedule has been set, the teachers and students can check the schedule of the courses from the user’s page. Then, teachers can access the course information page. Teachers can then also set the effort for their course’s objectives from the course information page (Figs. 6 and 7).

Fig. 6.

Schedule page

Fig. 7.

Page for setting the course objectives

After the efforts have been set, students can compare their own objectives with the objectives of the course (Fig. 8).

Fig. 8.

Course information page (1/2)

This feature of the course introduction module helps students determine which course they should take. In addition, teachers can also check the objectives of the participants of their courses, which helps them make teaching plans.

After the course has ended, the course information pages are changed (Fig. 9).

Fig. 9.

Course information page (2/2)

When students access the course information page, they can set the attainment data of each objective. They can also evaluate the courses in which they have taken part.

As shown in Fig. 10, lecturers also can check the attainments of the participants. This feature helps lecturers design better courses. Moreover, lecturers can set the credit data.

Fig. 10.

Course information page for lecturers

Based on the set of factors presented above, the administrators can determine the recommendation data.

As shown in Fig. 11, we can select from two methods: user-based or item-based methods. This method is almost the same as that of the drill module [4].

Fig. 11.

Recommendation setting page

After the recommendations are created, the recommended courses are displayed for students.

As shown in Fig. 12, a list of recommended courses is shown on the students’ page.

Fig. 12.

Recommendations on the student’s page

3 Method

This section describes a course for which the module was implemented and the resulting statistical analysis. First, we present the research question (RQ). We then introduce the course that provided the data for this research. Finally, the statistical analysis is presented.

3.1 Research Question and Hypothesis

As previously explained, the module allows students to compare objectives. Therefore, the module should help students to prepare for a course and ensure that the course is what the students anticipated it would be. Consequently, when the module works well, students tend not to be absent from a course or drop it.

Therefore, the following RQ and subsequent hypothesis is posited.

• RQ: How does the module affect student attendance rates?

• Hypothesis: The users of the module attend lectures more frequently than do students who do not use the module.

3.2 Course Characteristic

Here, we describe the characteristics of the course in which the module implemented, which are almost the same as those of the course described in [4].

As shown in Table 1, the course is an introduction to informatics for participants who do not major in informatics. The course was designed to help the class pass a qualifying examination on information technology. The number of attendees is similar to that in [4]. About 30% of attendees had previously used the course introduction module. Skewness and kurtosis of attendance rate are below 2.0. Hence, we can deal with the data as a normal distribution.

Table 1. Details of the course in which the module implemented

3.3 Statistical Analysis

Here we show the statistical analysis.

As shown in Table 2, the average attendance rate of the students who did not use the module is 68.9%. The users of the module attended 83.6% of lectures on average. Subsequently, students attended lectures more frequently when they used the module. According to the t-test, the difference is statistically significant (t(188.1) = −4.80, p < 0.01). Therefore, we can conclude that the hypothesis is confirmed.

Table 2. Average attendance rates

4 Discussion

As shown above, the module seems to have the effect that was expected. Namely, the users of the module attended lectures more frequently. However, it would be better if the module had some positive effect on examination scores. Theoretically, the module does not have a direct effect, and no such direct effect has been confirmed. As a next step, we will look for a better way of using the module to enable it to have a direct effect on student learning.

This paper have only confirmed the effect of the whole course introduction module. But we still need to confirm the effect of the recommendation engine of the module independently, although it is difficult to check this effect precisely. We must hence first determine a convincing method that will enable us to check the effect of the course recommendation engine. To check the effect of such recommendation engine may give us a hint to take advantage of big-data for education, because recommendation engine is one of the most important use of big-data.