skip to main content
10.1145/2910925.2910950acmotherconferencesArticle/Chapter ViewAbstractPublication PageswccceConference Proceedingsconference-collections
research-article

Measuring the Difficulty of Test Items in Computing Science Education

Published: 06 May 2016 Publication History

Abstract

This paper describes a communication-based approach for measuring the difficulty of test items used in computing science education, specifically, in first course in computer programming language. The assessment process is viewed here as a bi-directional and cyclical communication process, which consists of participants assuming the roles of senders and receivers, who are sending messages through channels. Thus, the difficulty of test items can be described from four perspectives: (1) message sender (test designer/teacher), (2) message receiver (student), (3) message (set of linguistic components), and (4) channel (medium for the message). These perspectives and related types of measurements were explored in an empirical study, and the preliminary results are presented in this paper. Analysis of the results shows that a communication-centered approach offers a broader view of the assessment process, reveals its multi-dimensionality, and captures new aspects of the tests' difficulty -- for example, time constraints.

References

[1]
Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J. and Wittrock, M.C., Eds. 2001. A taxonomy for learning and teaching and assessing: A revision of Bloom's taxonomy of educational objectives. Addison Wesley Longman, Inc.
[2]
Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H. and Krathwohl, D.R. 1956. Taxonomy of Educational Objectives: Handbook 1: Cognitive Domain. Longmans, Green and Co Ltd, London.
[3]
Biggs, J.B. and Collis, K.F. 1982. Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome). Academic Press, New York.
[4]
Fuller, U., Johnson, C. G., Ahoniemi, T., Cukierman, D., Hernán-Losada, I., Jackova, J., Lahtinen, E., Lewis, T. L. McGee Thompson, D., Riedesel, C. and Thompson E. 2007. Developing a computer science-specific learning taxonomy. ACM SIGCSE Bull. 39 (4): 152 --170.
[5]
Gunel, K. and Asliyan, R. 2009. Determining Difficulty of Questions in Intelligent Tutoring Systems. The Turkish Online Journal of Educational Technology. 8 (3) (July 2009), 14--21.
[6]
Haladyna, T.M. & Downing, S.M. 1993. How many options is enough for a multiple choice test item? Educational & Psychological Measurement, 53(4), 999--1010.
[7]
Kasto, N. and Whalley J.2013. Measuring the difficulty of code comprehension tasks using software metrics. In Proceedings of the Fifteenth Australasian Computing Education Conference (ACE2013), Adelaide, Australia, 59--65.
[8]
Krathwohl, D. R. 2002. A revision of Bloom's taxonomy: An overview. Theory into practice 41.4 (2002): 212--218.
[9]
Lister, R. 2001. Objectives and objective assessment in CS1. In ACM SIGCSE Bull. 33 (1): 292--296. ACM.
[10]
Nuttall, C. 2005. Teaching Reading Skills in a Foreign Language, Third Edition. Macmillan Books for Teachers.
[11]
Stiggins, R. J., & Bridgeford, N. J. 1985. The ecology of classroom assessment. Journal of Educational Measurement, 22(4), 271--286. http://dx.doi.org/10.1111/j.1745-3984.1985.tb01064
[12]
Traynor, D. and Gibson, J.P. 2005. Synthesis and analysis of automatic assessment methods in CS1: generating intelligent MCQs. In Proceedings of the 36th SIGCSE technical symposium on Computer science education (SIGCSE '05). ACM, New York, NY, USA, 495--499. DOI=http://dx.doi.org/10.1145/1047344.1047502
[13]
Whalley, J., Lister, R., Thompson, E., Clear, T., Robbins, P. and Prasad, C. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, using the Bloom and SOLO Taxonomies. Australian Computer Science Communications, 52: 243--252.

Cited By

View all
  • (2020)Evaluation of the Fight BAC! The Story of Your Dinner Campaign Video: A Multistate StudyJournal of Food Protection10.4315/0362-028X.JFP-19-32983:4(584-598)Online publication date: 9-Mar-2020
  • (2018)Multiple-Choice Questions in Programming CoursesACM Transactions on Computing Education10.1145/324313719:1(1-16)Online publication date: 19-Nov-2018

Index Terms

  1. Measuring the Difficulty of Test Items in Computing Science Education

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    WCCCE '16: Proceedings of the 21st Western Canadian Conference on Computing Education
    May 2016
    137 pages
    ISBN:9781450343558
    DOI:10.1145/2910925
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 May 2016

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Assessment
    2. difficulty level
    3. measurements

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    WCCCE '16

    Acceptance Rates

    WCCCE '16 Paper Acceptance Rate 26 of 35 submissions, 74%;
    Overall Acceptance Rate 78 of 117 submissions, 67%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)10
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2020)Evaluation of the Fight BAC! The Story of Your Dinner Campaign Video: A Multistate StudyJournal of Food Protection10.4315/0362-028X.JFP-19-32983:4(584-598)Online publication date: 9-Mar-2020
    • (2018)Multiple-Choice Questions in Programming CoursesACM Transactions on Computing Education10.1145/324313719:1(1-16)Online publication date: 19-Nov-2018

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media