Exam paper generation based on performance prediction of student group

Abstract

Exam paper generation is an indispensable part of teaching. Existing methods focus on the use of question extraction algorithms with labels for each question provided. Obviously, manual labeling is inefficient and cannot avoid label bias. Furthermore, the quality of the exam papers generated by the existing methods is not guaranteed. To address these problems, we propose a novel approach to generating exam papers based on prediction of exam performance. As such, we update the quality of the initially generated questions one by using dynamic programming, as well as in batches by using genetic algorithms. We performed the prediction task by using Deep Knowledge Tracing. Our approach considered the skill weight, difficulty, and distribution of exam scores. By comparisons, experimental results indicate that our approach performed better than the two baselines. Furthermore, it can generate exam papers with adaptive difficulties closely to the expected levels, and the related student exam scores will be guaranteed to be relatively reasonable distribution. In addition, our approach was evaluated in a real learning scenarios and shows advantages.

Introduction

The generation of exam questions is a challenging task in educational technology. The related research are roughly two categorizes. One is to use methods such as Natural Language Processing and Semantic Ontology to generate new questions from text or paragraphs [6], [28], and these methods focus on generating natural questions. Another is to extract questions from the question bank [13], [37], and the related methods consider the characteristics of the questions and their relevance to the student’s learning state. The task of exam paper generation (EPG) extracts several different questions from the question bank. Similarly, EPG depends on the characteristics of the questions, and needs to consider the students’ learning status.

EPG must consider various factors of an exam paper, such as its difficulty level, the coverage of assessed skills (synonyms of the knowledge points in this article), and the score of each question. Therefore, EPG is an optimization process with multiple objectives [8], [22]. However, with regard to translating the difficulty of each question into the difficulty of the entire exam paper, the existing methods do not propose a reasonable solution. Moreover, the exam is designed for the whole student group, rather than for a single student, and thus the difficulty is essentially a relative indicator. Therefore, different students may have different feelings about the difficulty level of the same question. Unfortunately, most existing EPG methods [36], [16], [24] rely on manually labeling the difficulty level of the question. This may be inefficient and produce label bias. Obviously, these methods cannot ensure that the difficulty of the generated exam paper is reasonable. Moreover, the existing EPG methods ignore an important issue that a good exam paper should be verified by the results of the exam [14], [10]. That is to say, the existing EPG methods cannot measure the quality of the exam papers generated by them in practical applications. Although some research has noticed the relationship between the EPG and the rationality of the exam results[12], there is still a long way to go before reaching the ideal goal.

In actual teaching activities, a common solution to EPG is to update an old version exam paper. The teacher can update the questions and adjust the difficulty of the exam paper based on his/her knowledge of students’ learning status. Constant adjustments make the distribution of skills and difficulty of the exam paper more and more reasonable. Inspired by this, we propose a novel exam paper generation approach based on performance prediction student groups. Where, the method of optimizing exam papers using dynamic programming named PDP-EG, and the other is using genetic algorithm named PGA-EG. In our study, skill is considered as the basic unit because it is the backbone embedded in various entities that appears during the learning process. Therefore, students’ skill mastery level determines whether they can correctly answer question related to the corresponding skill.

Our approach adopts Deep Knowledge Tracing [25] to achieve the students performance prediction task based on students’ exercise answered records. The exam paper generated by PDP-EG or PGA-EG could meet the difficulty level and skill weight requirements. The distribution of the achieved scores on our generated exam is close to that of requirement without manually setting the difficulty levels of the questions. The experimental results show that the exam papers generated by our approach are more advantageous than the baseline methods in terms of the main evaluation metrics of exam paper. In conclusion, the main contributions of our research are as follows:

• We propose a novel EPG approach that can generate the exam paper of a given difficulty without setting the difficult level of each question. In addition, the exam papers generated by our approach can ensure that the distribution of the achieved scores on exam are more reasonable.

• Our approach is able to well match the predicted student mastery levels of skills into the difficulty levels of the exam. This is achieved by using the multi-objective optimization algorithms.

• To the best of our knowledge, this is the first work on introducing the distribution of skill into EPG.

The organization of the paper is as follows: Section 2 starts with a focused review of some related work. Section 3 describes preliminaries and important notations. Sections 4 PDP-EG model, 5 PGA-EG model detail the proposed frameworks of PDP-EG and PGA-EG. Section 6 presents the setup of the experiments and results. And Section 7 draws conclusions for this paper.