Skip to main content
SpringerLink
Log in

An Experimental Evaluation of Automatically Generated Multiple Choice Questions from Ontologies

  • Conference paper
OWL: Experiences and Directions – Reasoner Evaluation (OWLED 2016, ORE 2016)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10161))

Abstract

In order to provide support for the construction of MCQs, there have been recent efforts to generate MCQs with controlled difficulty from OWL ontologies. Preliminary evaluation suggests that automatically generated questions are not field ready yet and highlight the need for further evaluations. In this study, we have presented an extensive evaluation of automatically generated MCQs. We found that even questions that adhere to guidelines are subject to the clustering of distractors. Hence, the clustering of distractors must be realised as this could affect the prediction of difficulty.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For a detailed description of both ontologies, the reader is referred to [1, 2].

  2. 2.

    Downloaded from: http://nlp.stanford.edu/software/tagger.shtml.

  3. 3.

    This is a grammatical correction that is manually added to the question.

References

  1. Alsubait, T., Parsia, B., Sattler, U.: Generating multiple choice questions from ontologies: lessons learnt. In: OWLED, Chicago, pp. 73–84 (2014)

    Google Scholar 

  2. Alsubait, T., Parsia, B., Sattler, U.: Generating multiple choice questions from ontologies: how far can we go? In: Lambrix, P., Hyvönen, E., Blomqvist, E., Presutti, V., Qi, G., Sattler, U., Ding, Y., Ghidini, C. (eds.) EKAW 2014. LNCS (LNAI), vol. 8982, pp. 66–79. Springer, Cham (2015). doi:10.1007/978-3-319-17966-7_7

    Chapter  Google Scholar 

  3. Haladyna, T.M., Downing, S.M., Rodriguez, M.C.: A review of multiple-choice item-writing guidelines for classroom assessment. Appl. Measur. Educ. 15(3), 309–333 (2002)

    Article  Google Scholar 

  4. Pho, V.-M., Andre, T., Ligozat, A.-L., Grau, B., Illouz, G., Francois, T., et al.: Multiple choice question corpus analysis for distractor characterization. In: LREC, pp. 4284–4291, Reykjavik (2014)

    Google Scholar 

Download references

Acknowledgments

The authors would like to thank Tahani Alsubait for sharing the MCQ generator code.

Author information

Authors and Affiliations

  1. School of Computer Science, The University of Manchester, Kilburn Building, Oxford Road, Manchester, M13 9PL, UK

    Ghader Kurdi, Bijan Parsia & Uli Sattler

Authors
  1. Ghader Kurdi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bijan Parsia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Uli Sattler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ghader Kurdi .

Editor information

Editors and Affiliations

  1. Fondazione Bruno Kessler, Povo, Trento, Italy

    Mauro Dragoni

  2. Universitad Politecnica de Madrid, Madrid, Spain

    María Poveda-Villalón

  3. University of Oslo, Oslo, Norway

    Ernesto Jimenez-Ruiz

Appendices

Appendix

A Question Categories

Table 12. Explanation of the six question categories generated by the MCQ generator (adapted from [1]).
Full size table

B Example Questions

Syntactic Clues

Examples of the form (SK)

  • Stem: State Transition Network ...:

    figure c

Examples of the form (SD)

  • Stem: Repertory Grid Stage 2 ...

    figure d

  • Stem: Which of the following terms can be defined by “a Java keyword used to declare a variable that holds an 8 bit signed integer”?

    figure e

Examples of the form (ANT)

  • Stem: Which of the following is [a]Footnote 3 Binary Operator?

    figure f

Syntactic Consistency

  • Stem: What is [a] Book?

    figure g

  • Stem: Which of the following is [a] Java Language Feature?

    figure h

Note that in the previous example, although (D) is inconsistent with the key, it is indeed a Java language feature.

  • Stem: Which of the following terms can be defined by “A binary remainder operator that produces a pure value that is the remainder from an implied division of its operands”?

    figure i

In this example, both distractors (A) and (B) are inconsistent with the key but they are both plausible. By investigating the ontology, we found that this issue resulted from the inconsistent naming of concepts.

Semantic homogeneity

  • Stem: Which of the following terms can be defined by “A layout manager that allows subcomponents to be added in up to five places specified by constants NORTH, SOUTH, EAST, WEST and CENTER”?

    figure j

It is clearly deduced from the previous question that the expected answer is a layout manager. As can be seen, distractors (A) and (B) are heterogeneous in relation to the key type while option (D) is homogeneous.

Clustered Distractors

  • Stem: Which of the following terms can be defined by “A stage in the software development process where customer needs are translated into how it could be implemented”?

    figure k

Distractors (A) and (B) are clustered since knowing that the answer is not testing will allow the elimination of all types of testing.

  • Stem: Protocol Analysis Technique ...

    figure l

  • Stem: Which of the following is produces some Protocol?

    figure m

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Kurdi, G., Parsia, B., Sattler, U. (2017). An Experimental Evaluation of Automatically Generated Multiple Choice Questions from Ontologies. In: Dragoni, M., Poveda-Villalón, M., Jimenez-Ruiz, E. (eds) OWL: Experiences and Directions – Reasoner Evaluation. OWLED ORE 2016 2016. Lecture Notes in Computer Science(), vol 10161. Springer, Cham. https://doi.org/10.1007/978-3-319-54627-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54627-8_3

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54626-1

  • Online ISBN: 978-3-319-54627-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation