Abstract
In this paper, we present an unsupervised dependency-based approach to extract semantic relations to be applied in the context of automatic generation of multiple choice questions (MCQs). MCQs also known as multiple choice tests provide a popular solution for large-scale assessments as they make it much easier for test-takers to take tests and for examiners to interpret their results. Manual generation of MCQs is a very expensive and time-consuming task and yet they often need to be produced on a large scale and within short iterative cycles. We approach the problem of automated MCQ generation with the help of unsupervised relation extraction, a technique used in a number of related natural language processing problems. The goal of Unsupervised relation extraction is to identify the most important named entities and terminology in a document and then recognise semantic relations between them, without any prior knowledge as to the semantic types of the relations or their specific linguistic realisation. We use these techniques to process instructional texts and identify those facts (terminology, entities, and semantic relations between them) that are likely to be important for assessing test-takers’ familiarity with the instructional material. We investigate an approach to learn semantic relations between named entities by employing a dependency tree model. Our findings show that an optimised configuration of our MCQ generation system is capable of attaining high precision rates, which are much more important than recall in the automatic generation of MCQs. We also carried out a user-centric evaluation of the system, where subject domain experts evaluated automatically generated MCQ items in terms of readability, usefulness of semantic relations, relevance, acceptability of questions and distractors and overall MCQ usability. The results of this evaluation make it possible for us to draw conclusions about the utility of the approach in practical e-Learning applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Afzal N, Mitkov R, Farzindar A (2011) Unsupervised Relation extraction using dependency trees for automatic generation of multiple-choice questions. In: Butz C, Lingras P (eds) Proceedings of the Canadian AI 2011, LNAI 6657. Springer, Heidelberg, pp 32–43
Agichtein E, Gravano L (2000) Snowball: Extracting Relations from Large Plaintext Collections. In: Proceedings of the 5th ACM international conference on digital libraries
Bikel DM, Miller S, Schwartz R, Weischedel R (1998) Nymble: a high-performance learning name-finder. In Proceedings of the conference on applied natural language processing
Brown J, Frishkoff G, Eskenazi M (2005) Automatic question generation for vocabulary assessment. In: Proceeding of HLT/EMNLP. Vancouver, BC
Caraballo SA (1999) Automatic construction of a hypernym-labeled noun hierarchy from text. In: Proceedings of 37th annual meeting of the association for computational linguistics, pp 120–126
Carlsson C, Brunelli M, Mezei J (2012) Decision making with a fuzzy ontology. Soft Comput 16(7):1143–1152
Chen C-Y, Liou H-C, Chang JS (2006) FAST—an automatic generation system for grammar tests. In: Proceedings of COLING/ACL interactive presentation sessions, Sydney
Chen W, Aist G, Mostow J (2009) Generating questions automatically from informational text. In: Proceedings of the 2nd workshop on question generation. Brighton
Cohen AM, Hersh WR (2005) A survey of current work in biomedical text mining. Brief Bioinform 6(1):57–71
Cohen J (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull
Corney DP, Jones D, Buxton B, Langdon W (2004) BioRAT: extracting biological information from full-length papers. Bioinformatics 20:3206–3213
Cover T, Thomas J (1991) Elements of information theory. Wiley, New York
Dagan I, Lee L, Pereira F (1997) Similarity-based methods for word sense disambiguation. In: Proceedings of the 35th annual meeting of the association for computational linguistics, Madrid, p 56.63
Dagan I, Lee L, Pereira F (1999) Similarity-based models of word cooccurrence probabilities. Mach Learn J 34(1–3):43–69
Das R, Elikkottil A (2010) Auto-summarizer to aid a Q/A system. Int J Comput Appl 1(1):113–117
De Maio C, Fenza G, Loia V, Senatore S (2009) Towards an automatic fuzzy ontology generation. In: Proceedings of IEEE international conference on fuzzy systems, pp 1044–1049
Dhillon IS, Mallela S, Kumar R (2002) Enhanced word clustering for hierarchical text classification (Tech. Rep. Nos. TR-02-17). Austin: Department of Computer Sciences, University of Texas
Farzindar A, Lapalme G (2004) LetSum, an automatic Legal Text Summarizing system. In: Gordon Thomas F (ed) Legal Knowledge and Information Systems, Jurix 2004: the 7th annual conference. IOS Press, Berlin, pp 11–18
Firth JR (1957) A synopsis of linguistic theory 1930–1955. Studies in Linguistic Analysis. Blackwell, Oxford, pp 1–32
Gates D (2008) Generating Look-Back Strategy Questions from Expository Texts. In: Workshop on the question generation shared task and evaluation challenge. NSF, Arlington
Graesser A, Person N (1994) Question asking during tutoring. Am Educ Res J 31:104–137
Graesser AC, Chipman P, Haynes BC, Olney A (2005) Autotutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans Educ 48(4):612–618
Grefenstette G (1994) Explorations in automatic Thesaurus discovery, vol. 278 of Kluwer International Series in Engineering and Computer Science. Kluwer, Boston
Gronlund N (1982) Constructing achievement tests. Prentice Hall, New York
Harris Z (1954) Distributional structure. Word 10(23):146–162
Harshman R (1970) Foundations of the parafac procedure: Models and conditions for an “explanatory” multi-modal factor analysis. In: UCLA Working Papers in Phonetics, vol 16
Hasegawa T, Sekine S, Grishman R (2004) Discovering relations among named entities from large corpora. In: Proceedings of ACL’04
Hatzivassiloglou V (1996) Do we need linguistics when we have statistics? A comparative analysis of the contributions of linguistic cues to a statistical word grouping system. In: Judith K, Philip R (eds) The balancing act: combining symbolic and statistical approaches to language, chapter 4. MIT Press, Cambridge, pp 67–94
Hirschman L, Mani I (2003) Evaluation. In: Mitkov R (ed) The Oxford Handbook of Computational Linguistics. Oxford University Press, UK, pp 414–429
Hodges PE, McKee AH, Davis BP, Payne WE, Garrels JI (1999) The Yeast Proteome Database (YPD): a model for the organization and presentation of genomewide functional data. Nucleic Acids Res 27(1): 69–73
Hoshino A, Nakagawa H (2007) Assisting cloze test making with a web application. In: Proceedings of society for information technology and teacher education international conference, Chesapeake
Huang M, Zhu X, Payan GD, Qu K, Li M (2004) Discovering patterns to extract protein-protein interactions from full biomedical texts. Bioinformatics, pp 3604–3612
Kalady S, Elikkottil A, Das R (2010) Natural language question generation using syntax and keywords. In: Proceedings of the 3rd workshop on question generation
Karamanis N, Ha LA, Mitkov R (2006) Generating multiple-choice test items from medical text: A pilot study. In: Proceedingd of the 4th international natural language generation conference, (July), pp 111–113
Kullback S, Leibler R (1951) On information and sufficiency. Ann Math Stat 22:79–86
Lapata M, Keller F, McDonald S (2001) Evaluating smoothing algorithms against plausibility judgements. In: Proceedings of the 39th annual meeting of the association for computational linguistics (ACL-2001), Toulouse, pp 346–353
Lin D (1998) Automatic retrieval and clustering of similar words. In: Proceedings of international conference on computational linguistics and the annual meeting of the association for Computational Linguistics
Lin J (1991) Divergence measures based on the Shannon entropy. IEEE Trans Inform Theory 37(1):145–151
Martin EP, Bremer E, Guerin G, DeSesa M-C, Jouve O (2004) Analysis of protein/protein interactions through biomedical literature: text mining of abstracts vs. text mining of full text articles. Springer, Berlin, pp 96–108
Mitkov R, An LA (2003) Computer-aided generation of multiple-choice tests. In: Proceedings of the HLT/NAACL 2003 workshop on building educational applications using natural language processing, Edmonton, pp 17–22
Mitkov R, Ha LA, Karamanis N (2006) A computer-aided environment for generating multiple-choice test items. Natural Language Engineering 12(2). Cambridge University Press, Cambridge, pp 177–194
Mostow J, Chen W Generating Instruction Automatically for the Reading Strategy of Self-Questioning. In: Proceedings of the 14th international conference on artificial intelligence in Education, Brighton
Nielsen R (2008) Question generation: Proposed challenge tasks and their evaluation. In: Proceedings of the workshop on the question generation shared task and evaluation, challenge
Palmer M, Kingsbury P, Gildea D (2005) The proposition bank: an annotated corpus of semantic roles. Comput Linguist 31(1): 71–106
Papasalouros A, Kanaris K, Konstantinos K (2008) Automatic generation of multiple choice questions from domain ontologies. In: Proceeding of IADIS international conference e-learning
Paroubek P, Chaudiron S, Hirschman L (2007) Principles of evaluation in natural language processing. TAL 48(1/2007):7–31
Pereira F, Tishby N, Lee L (1993) Distributional clustering of similar words. In: Proceedings of the 31st annual meeting of the association for computational linguistics (ACL-1993), Columbus, pp 183–190
Pradhan S, Hacioglu K, Krugler V, Ward W, Martin JH, Jurafsky D (2005) Support vector learning for semantic argument classification. Mach Learn 60(1):11–39
Rao CR (1983) Diversity: its measurement, decomposition, apportionment and analysis. Indian J Stat 44(A):1–22
Schwartz L, Aikawa T, Pahud M (2004) Dynamic language learning tools. In: Proceedings of the of the 2004 In-STIL/ICALL Symposium
Stevenson M, Greenwood M (2005) A semantic approach to IE pattern induction. In: Proceedings of ACL’05, pp 379–386
Stevenson M, Greenwood M (2009) Dependency pattern models for information extraction. Res Lang Comput
Sumita E, Sugaya F, Yamamoto S (2005) Measuring non-native speakers’ proficiency of English using a test with automatically-generated fill-in-the-blank questions. In: Proceedings of the 2nd workshop on building educational applications using NLP, pp 61–68
Tateno J, Sano H, Aizawa H, Nakamura T, Morita Y (2005) Producing english Educational materials form the BNC and releasing them on the Web, IEICE Technical report, TL2005-1826, Tokyo, pp 7–12
Ureel L, Forbus K, Riesbeck C, Birnbaum L (2005) Question generation for learning by reading. In: Proceedings of the AAAI workshop on textual question answering, Pittsburgh
Vanderwende L (2007) Answering and questioning for machine reading. In: Proceedings of the 2007 AAAI spring symposium on machine reading, Stanford
Vanderwende L (2008) The importance of being important: question generation. In: Proceedings of the workshop on the question generation shared task and evaluation challenge, Arlington
Walker MA, Rambow O, Rogati M (2001) Spot: a trainable sentence planner. In: Proceedings of NAACL
Weeds J (2003) Measures and applications of lexical distributional similarity. Ph.D. thesis, University of Sussex
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by V. Loia.
Rights and permissions
About this article
Cite this article
Afzal, N., Mitkov, R. Automatic generation of multiple choice questions using dependency-based semantic relations. Soft Comput 18, 1269–1281 (2014). https://doi.org/10.1007/s00500-013-1141-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-013-1141-4