A novel approach to generate MCQs from domain ontology: Considering DL semantics and open-world assumption

Abstract

Ontologies are structures, used for knowledge representation, which model domain knowledge in the form of concepts, roles, instances and their relationships. This knowledge can be exploited by an assessment system in the form of multiple choice questions (MCQs). The existing approaches, which use ontologies expressed in the Web Ontology Language (OWL) for MCQ generation, are limited to simple concept related questions — “What is C?” or “Which of the following is an example of C?” (where $C$ is a concept symbol) — or analogy type questions involving roles. There are no efforts in the literature which make use of the terminological axioms in the ontology such as existential, universal and cardinality restrictions on concepts and roles for MCQ generation. Also, there are no systematic methods for generating incorrect answers (distractors) from ontologies. Distractor generation process has to be given much importance, since the generated distractors determine the quality and hardness of an MCQ. We propose two new MCQ generation approaches, which generate MCQs that are very useful and realistic in conducting assessment tests, and the corresponding distractor generating techniques. Our distractor generation techniques, unlike other methods, consider the open-world assumption, so that the generated MCQs will always be valid (falsity of distractors is ensured). Furthermore, we present a measure to determine the difficulty level (a value between 0 and 1) of the generated MCQs. The proposed system is implemented, and experiments on specific ontologies have shown the effectiveness of the approaches. We also did an empirical study by generating question items from a real-world ontology and validated our results with the help of domain experts.

Introduction

Automated assessment systems serve as a method to measure the level of learning as well as to provide a faster solution for large scale assessments. Many tests like TOEFL, IELTS, GRE ​and GMAT are dependent on online assessment systems to make the assessment task easier. Such systems mainly use multiple choice questions rather than subjective questions for conducting the test.

Using Multiple Choice Questions (MCQs) for assessments has both merits and demerits. They are preferred for assessing broad range of knowledge. This is mainly because they require less administrative overhead and provide instant feedback to test takers. However, studies by Barbara Gross  [1] and Sidick et al.  [2] show that, developing effective objective type questions is time consuming and requires domain expertise to generate good quality MCQs. So, there is a need for an automated method for MCQ generation from a given knowledge source.

Recently, a handful of studies [3], [4], [5], [6], [7], [8], [9] explored the use of structured domain knowledge in the form of description logic based ontologies to automatically generate MCQs. This would enable online assessment systems to utilize existing knowledge bases for the assessment of learner’s knowledge and skills. But, there are challenges involved in generating MCQs from these ontologies. Some of the challenges that the existing approaches tried to address are: (i) How to frame interesting and good quality questions from ontologies? (ii) How to generate proper incorrect answers (distractors) for the framed question? (iii) How to control the difficulty level of the generated questions? Although the previous efforts were not in vain, there are substantial shortcomings in fully exploiting the formalized knowledge in an ontology for MCQ generation. In this paper, we show that, with a better understanding of the semantics of a given ontology (expressed in Web Ontology Language), the three challenges can be addressed more elegantly.

Challenge 1. Framing interesting and good quality questions. In the literature, the approaches that use ontologies have the limitation that they generate simple concept related questions–“What is C?” or “Which of the following is an example of C?” (where $C$ is a concept symbol)–or analogy type questions involving roles. These questions are very basic  [10] and do not contain any domain related specifics. In other words, the approaches which generate such questions, do not appropriately make use of the axiomatized knowledge in an ontology. Furthermore, restrictions (existential, universal and cardinality) on concepts and roles in ontologies are not utilized properly for question generation in any of the current approaches.

Consider a movie ontology with statements,

With respect to this, we can frame a question about the instance braveHeart: “Choose a movie directed by an actor?” Our approach in this paper is an effort in this direction.

Challenge 2. Proper distractor generation. Under the closed-world assumption (CWA), we can choose any instance which is different from the instance braveHeart as a distractor for the question in the example above. But, Web Ontology Language (OWL) adheres to the open-world assumption (OWA): statements which are not logical consequences of a given knowledge base are not necessarily considered false. Therefore, not all distractors which are generated under CWA can be guaranteed as true distractors.

We observed that most of the existing MCQ generation techniques [7] randomly select instances which do not belong to the class of the correct answer as distractors. The incorrectness of the distractors cannot be ensured by this random selection method, which in turn made it necessary to manually check the correctness of the question items before making use of them. We address this issue by proposing a systematic method to generate only those question items which are valid under OWA.

Challenge 3. Control the difficulty level of the generated MCQ. MCQs of varying difficulty level are necessary to assess the depth of knowledge of a learner (student). We introduce a measure to find out the difficulty level of the generated MCQs based on the similarity-based theory suggested by Alsubait et al.  [11].

In this paper, we propose two approaches (i) node-label-set based approach (ii) edge-label-set based approach to generate (two) interesting types of MCQs. We adopt description logic specifications of the ontology to generate the so called label-sets (node-label-sets and edge-label-sets). A measure to estimate the difficulty level of generated MCQs is also proposed by means of these label-sets. We study the feasibility of our approaches by implementing them and generating MCQs from some sample ontologies. In  Appendix A, we list some of the MCQs, which are generated from Geographical Entity ontology. To validate our new approaches and difficulty measure, we generated question items from a real-world ontology and got them evaluated by domain experts. Statistics of our empirical evaluation validate our arguments and are detailed in Section  6. The new notations and abbreviations that we introduced in this paper are listed in  Appendix B along with their meaning.