Cited By
View all- Tam RGifford JBeck K(2022)Recent Developments in the Assessment of Nutrition Knowledge in AthletesCurrent Nutrition Reports10.1007/s13668-022-00397-111:2(241-252)Online publication date: 16-Feb-2022
The Impact of Multiple Choice Question Design on Predictions of Performance
Pages 66 - 72
Abstract
Multiple choice questions (MCQs) are a popular question format in introductory programming courses as they are a convenient means to provide scalable assessments. However, with typically only four or five answer options and a single correct answer, MCQs are prone to guessing and may lead students into a false sense of confidence. One approach to mitigate this problem is the use of Multiple-Answer MCQs (MAMCQs), where more than one answer option may be correct. This provides a larger solution space and may help students form more accurate assessments of their knowledge. We explore the use of this question format on an exam in a very large introductory programming course. The exam consisted of both MCQ and MAMCQ sections, and students were invited to predict their scores for each section. In addition, students were asked to report their preference for question format. We found that students over predicted their score on the MCQ section to a greater extent and that these prediction errors were more pronounced amongst less capable students. Interestingly, we found that students did not have a strong preference for MCQs over MAMCQs, and we recommend broader adoption of the latter format.
References
[1]
Pedro Henriques Abreu, Daniel Castro Silva, and Anabela Gomes. 2018. Multiple-Choice Questions in Programming Courses: Can We Use Them and Are Students Motivated by Them?ACM Transactions on Computing Education (TOCE) 19, 1 (2018), 1–16.
[2]
Benjamin S Bloom, MD Englehart, Edward J Furst, Walker H Hill, and David R Krathwohl. 1956. Taxonomy of educational objectives: Handbook I. Cognitive domain. New York: David McKay(1956).
[3]
Cynthia J. Brame. 2020. Team-based learning. Vanderbilt University. Retrieved October 12, 2020 from https://cft.vanderbilt.edu/guides-sub-pages/team-based-learning/
[4]
Richard F. Burton. 2004. Multiple choice and true/false tests: reliability measures and some implications of negative marking. Assessment & Evaluation in Higher Education 29, 5 (2004), 585–595. https://doi.org/10.1080/02602930410001689153 arXiv:https://doi.org/10.1080/02602930410001689153
[5]
Donald Chinn, Michael de Raadt, Anne Philpott, Judy Sheard, Mikko-Jussi Laakso, Daryl D’Souza, James Skene, Angela Carbone, Tony Clear, Raymond Lister, 2012. Introductory programming: examining the exams. In Proceedings of the Fourteenth Australasian Computing Education Conference-Volume 123. 61–70.
[6]
David Clark. 2004. Testing Programming Skills with Multiple Choice Questions. Informatics in Education 3, 2 (2004), 161–178. https://doi.org/10.15388/infedu.2004.12
[7]
Julie Considine, Mari Botti, and Shane Thomas. 2005. Design, format, validity and reliability of multiple choice questions for use in nursing research and education. Collegian 12, 1 (2005), 19 – 24. https://doi.org/10.1016/S1322-7696(08)60478-3
[8]
Paul Denny, John Hamer, Andrew Luxton-Reilly, and Helen Purchase. 2008. PeerWise: students sharing their multiple choice questions. In Proceedings of the fourth international workshop on computing education research. 51–58.
[9]
Paul Denny, Andrew Luxton-Reilly, John Hamer, Dana B. Dahlstrom, and Helen C. Purchase. 2010. Self-Predicted and Actual Performance in an Introductory Programming Course. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education (Bilkent, Ankara, Turkey) (ITiCSE ’10). Association for Computing Machinery, New York, NY, USA, 118–122. https://doi.org/10.1145/1822090.1822124
[10]
Paul Denny, Sathiamoorthy Manoharan, Ulrich Speidel, Giovanni Russello, and Angela Chang. 2019. On the Fairness of Multiple-Variant Multiple-Choice Examinations. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 462–468. https://doi.org/10.1145/3287324.3287357
[11]
Paul Denny, Ewan Tempero, Dawn Garbett, and Andrew Petersen. 2017. Examining a Student-Generated Question Activity Using Random Topic Assignment. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (Bologna, Italy) (ITiCSE ’17). Association for Computing Machinery, New York, NY, USA, 146–151. https://doi.org/10.1145/3059009.3059033
[12]
Steven Downing. 2005. The Effects of Violating Standard Item Writing Principles on Tests and Students: The Consequences of Using Flawed Test Items on Achievement Examinations in Medical Education. Advances in health sciences education : theory and practice 10 (02 2005), 133–43. https://doi.org/10.1007/s10459-004-4019-5
[13]
George T Duncan and EO Milton. 1978. Multiple-answer multiple-choice test items: Responding and scoring through bayes and minimax strategies. Psychometrika 43, 1 (1978), 43–57.
[14]
Brian Harrington, Shichong Peng, Xiaomeng Jin, and Minhaz Khan. 2018. Gender, Confidence, and Mark Prediction in CS Examinations. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education(Larnaca, Cyprus) (ITiCSE 2018). Association for Computing Machinery, New York, NY, USA, 230–235. https://doi.org/10.1145/3197091.3197116
[15]
Nicole Ibbett and Brett Wheldon. 2016. The Incidence of Clueing in Multiple Choice Testbank Questions in Accounting: Some Evidence from Australia. e-Journal of Business Education and Scholarship of Teaching 10, 1(2016), 20–35.
[16]
Meera Komarraju and Dustin Nadler. 2013. Self-efficacy and academic achievement: Why do implicit beliefs, goals, and effort regulation matter?Learning and individual differences 25 (2013), 67–72.
[17]
Justin Kruger and David Dunning. 1999. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments.Journal of personality and social psychology 77, 6(1999), 1121.
[18]
Justin Kruger and David Dunning. 2000. Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments. Journal of Personality and Social Psychology 77 (01 2000), 1121–34. https://doi.org/10.1037//0022-3514.77.6.1121
[19]
William L Kuechler and Mark G Simkin. 2003. How well do multiple choice tests evaluate student understanding in computer programming classes?Journal of Information Systems Education 14, 4 (2003), 389.
[20]
Raymond Lister. 2000. On Blooming First Year Programming, and Its Blooming Assessment. In Proceedings of the Australasian Conference on Computing Education (Melbourne, Australia) (ACSE ’00). Association for Computing Machinery, New York, NY, USA, 158–162. https://doi.org/10.1145/359369.359393
[21]
Raymond Lister. 2001. Objectives and Objective Assessment in CS1. In Proceedings of the Thirty-Second SIGCSE Technical Symposium on Computer Science Education (Charlotte, North Carolina, USA) (SIGCSE ’01). Association for Computing Machinery, New York, NY, USA, 292–296. https://doi.org/10.1145/364447.364605
[22]
P McCoubrie and L McKnight. 2008. Single best answer MCQs: a new format for the FRCR part 2a exam. Clinical Radiology 63, 5 (2008), 506–510.
[23]
David Nicol. 2007. E‐assessment by design: using multiple‐choice tests to good effect. Journal of Further and Higher Education 31, 1 (2007), 53–64.
[24]
Andrew Petersen, Michelle Craig, and Paul Denny. 2016. Employing Multiple-Answer Multiple Choice Questions. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education(Arequipa, Peru) (ITiCSE ’16). Association for Computing Machinery, New York, NY, USA, 252–253. https://doi.org/10.1145/2899415.2925503
[25]
Andrew Petersen, Michelle Craig, and Daniel Zingaro. 2011. Reviewing CS1 Exam Question Content. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (Dallas, TX, USA) (SIGCSE ’11). Association for Computing Machinery, New York, NY, USA, 631–636. https://doi.org/10.1145/1953163.1953340
[26]
Tim S. Roberts. 2006. The Use of Multiple Choice Tests for Formative and Summative Assessment. In Proceedings of the 8th Australasian Conference on Computing Education - Volume 52 (Hobart, Australia) (ACE ’06). Australian Computer Society, Inc., AUS, 175–180.
[27]
Karim Sadeghi and Ghazal Akhavan Masoumi. 2017. Does number of options in multiple choice tests affect item facility and discrimination? An examination of test-taker preferences. Journal of English Language Teaching and Learning 9, 19 (2017), 123–143.
[28]
Shuhaida Shuhidan, Margaret Hamilton, and Daryl D’Souza. 2010. Instructor perspectives of multiple-choice questions in summative assessment for novice programmers. Computer Science Education 20, 3 (2010), 229–259.
[29]
Simon. 2011. Wrong is a Relative Concept: Part Marks for Multiple-Choice Questions. In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114 (Perth, Australia) (ACE ’11). Australian Computer Society, Inc., AUS, 47–54.
[30]
Simon, Judy Sheard, Daryl D’Souza, Mike Lopez, Andrew Luxton-Reilly, Iwan Handoyo Putro, Phil Robbins, Donna Teague, and Jacqueline Whalley. 2015. How (not) to write an introductory programming exam. In Proceedings of the 17th Australasian Computing Education Conference (ACE 2015). Australian Computer Society, Inc., Sydney, Australia, 137–146.
[31]
Simon and Susan Snowdon. 2014. Multiple-Choice vs Free-Text Code-Explaining Examination Questions. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research(Koli, Finland) (Koli Calling ’14). Association for Computing Machinery, New York, NY, USA, 91–97. https://doi.org/10.1145/2674683.2674701
[32]
Karyn Woodford and Peter Bancroft. 2005. Multiple Choice Questions Not Considered Harmful. In Proceedings of the 7th Australasian Conference on Computing Education - Volume 42 (Newcastle, New South Wales, Australia) (ACE ’05). Australian Computer Society, Inc., AUS, 109–116.
[33]
Moshe Zeidner. 1987. Essay versus Multiple-Choice Type Classroom Exams: The Student’s Perspective. The Journal of Educational Research 80, 6 (1987), 352–358. https://doi.org/10.1080/00220671.1987.10885782 arXiv:https://doi.org/10.1080/00220671.1987.10885782
Index Terms
- The Impact of Multiple Choice Question Design on Predictions of Performance
Index terms have been assigned to the content through auto-classification.
Recommendations
Employing Multiple-Answer Multiple Choice Questions
ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science EducationIncreasing enrollments and adoption of online resources have encouraged the use of multiple choice questions as a means of providing scalable assessment. However, in contexts where formative feedback is desired, standard multiple choice questions may ...
Multiple choice questions not considered harmful
ACE '05: Proceedings of the 7th Australasian conference on Computing education - Volume 42Increasingly, academics are confronted with issues associated with assessment in large classes, arising from a combination of factors including higher student enrolments and the introduction of a trimester of study in many universities. The resulting ...
Multiple Choice Questions with Justifications
T4E '14: Proceedings of the 2014 IEEE Sixth International Conference on Technology for EducationMultiple choice questions (MCQs) are widely used as an efficient means to grade large batches of students. With technology enabling extremely large classes (MOOCs), the use of MCQs has increased rapidly, leading to an increased scrutiny of their ...
Comments
Information & Contributors
Information
Published In
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 March 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ACE '21
Acceptance Rates
Overall Acceptance Rate 161 of 359 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 112Total Downloads
- Downloads (Last 12 months)22
- Downloads (Last 6 weeks)2
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Tam RGifford JBeck K(2022)Recent Developments in the Assessment of Nutrition Knowledge in AthletesCurrent Nutrition Reports10.1007/s13668-022-00397-111:2(241-252)Online publication date: 16-Feb-2022
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format