Kate Sanders
Judy Sheard
Brett A. Becker
ABSTRACT
The goal of most computing education research is to effect positive change in how computing is taught and learned. Statistical techniques are one important tool for achieving this goal. In this paper we report on an analysis of ICER papers that use inferential statistics. We present the most commonly used techniques; an overview of the techniques the ICER community has used over its first 14 years of papers, grouped according to the purpose of the technique; and a detailed analysis of three of the most commonly used techniques (t-test, chi-squared test, and Mann-Whitney-Wilcoxon). We identify common flaws in reporting and give examples of papers where statistics are reported well. In sum, the paper draws a picture of the use of inferential statistics by the ICER community. This picture is intended to help orient researchers who are new to the use of statistics in computing education research and to encourage reflection by the ICER community on how it uses statistics and how it can improve that use.
- American Educational Research Association. 2006. Standards for Reporting on Empirical Social Science Research in AERA Publications. American Educational Research Association. http://www.aera.net/Portals/38/docs/12ERv35n6_Standard4Report.pdf, (accessed 1 April 1 2019).Google Scholar
- American Psychological Association. 2009. Publication Manual 6th ed.). American Psychological Association, Washington, DC.Google Scholar
- Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, and Debajyoti Sinha. 1999. Beyond kappa: a review of interrater agreement measures. Canadian Journal of Statistics, Vol. 27, 1 (1999), 3--23.Google ScholarCross Ref
- Lisa Bender, Gursimran Walia, Krishna Kambhampaty, Kendall E Nygard, and Travis E Nygard. 2012. Social sensitivity correlations with the effectiveness of team process performance: an empirical study. In {Eighth} International Computing Education Research Conference (ICER 2012). ACM, New York, NY, USA, 39--46. Google ScholarDigital Library
- Harris M Cooper. 1988. Organizing knowledge syntheses: a taxonomy of literature reviews. Knowledge in Society, Vol. 1, 1 (1988), 104.Google Scholar
- Stephen Cooper, Karen Wang, Maya Israni, and Sheryl Sorby. 2015. Spatial skills training in introductory computing. 11th International Computing Education Research Conference (ICER 2015). ACM, 13--20. Google ScholarDigital Library
- Mark Davies and Joseph L Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, Vol. 38, 4 (1982), 1047--1051. http://www.jstor.org/stable/2529886Google ScholarCross Ref
- Sally Fincher and Marian Petre (Eds.). 2004. Computer Science Education Research. Taylor & Francis, London. Google ScholarDigital Library
- Brian Hanks. 2007. Problems encountered by novice pair programmers. Third International Computing Education Research Workshop (ICER 2007). 159--164. Google ScholarDigital Library
- Christopher D Hundhausen and Jonathan Lee Brown. 2005. Personalizing and discussing algorithms within CS1 studio experiences: an observational study. In First International Computing Education Research Workshop (ICER 2005). 45--56. Google ScholarDigital Library
- Matthew C Jadud and Brian Dorn. 2015. Aggregate compilation behavior: findings and implications from 27,698 users. In 11th International Computing Education Research Conference (ICER 2015). 131--139. Google ScholarDigital Library
- Ximena López, Jorge Valenzuela, Miguel Nussbaum, and Chin-Chung Tsai. 2015. Some recommendations for the reporting of quantitative studies. Computers & Education, Vol. 91 (2015), 106--110. Google ScholarDigital Library
- Lauri Malmi, Judy Sheard, Simon, Roman Bednarik, Juha Helminen, Ari Korhonen, Niko Myller, Juha Sorva, and Ahmad Taherkhani. 2010. Characterizing research in computing education: a preliminary analysis of the literature. In Sixth International Computing Education Research Workshop (ICER 2010). 3--12. Google ScholarDigital Library
- Joe Miró Julià, David López, and Ricardo Alberich. 2012. Education and research: evidence of a dual life. {Eighth} International Computing Education Research Conference (ICER 2012). 17--22. Google ScholarDigital Library
- Briana B Morrison, Brian Dorn, and Mark Guzdial. 2014. Measuring cognitive load in introductory CS: adaptation of an instrument. In Tenth International Computing Education Research Conference (ICER 2014). 131--138. Google ScholarDigital Library
- Kimberly A Neuendorf. 2016. The Content Analysis Guidebook. Sage.Google Scholar
- Elizabeth Patitsas, Jesse Berlin, Michelle Craig, and Steve Easterbrook. 2016. Evidence that computer science grades are not bimodal. In 12th International Computing Education Research Conference (ICER 2016). 113--121. Google ScholarDigital Library
- JJ Randolph. 2007 a. Computer science education research at the crossroads. A methodological review of computer science education research: 2000--2005 . Ph.D. Dissertation. Utah State University. https://archive.org/details/randolph_dissertation (accessed 15 February 2019). Google ScholarDigital Library
- JJ Randolph. 2007 b. Findings from "A methodological review of the computer science education research: 2000--2005". SIGCSE Bulletin, Vol. 39, 4 (Dec. 2007), 130. Google ScholarDigital Library
- JJ Randolph. 2009. A guide to writing the dissertation literature review. Practical Assessment, Research & Evaluation, Vol. 14, 13 (2009), 1--13.Google Scholar
- JJ Randolph, R Bednarik, and N Myller. 2005. A methodological review of the articles published in the proceedings of Koli Calling 2001--2004. In Koli Calling 2005 Conference on Computer Science Education (Koli Calling 2005). 103--109. Linked from https://www.kolicalling.fi/index.php/previous-koli-calling-conferences (Retrieved 15 February 2019).Google Scholar
- JJ Randolph, G Julnes, E Sutinen, and S Lehman. 2008. A methodological review of computer science education research. Journal of Information Technology Education, Vol. 7 (2008), 135--162.Google ScholarCross Ref
- Judy Sheard, Simon, Margaret Hamilton, and Jan Lönnberg. 2009. Analysis of research into the teaching and learning of programming. In Fifth International Computing Education Research Workshop (ICER 2009). 93--104. Google ScholarDigital Library
- Simon. 2007 a. A classification of recent Australasian computing education publications. Computer Science Education, Vol. 17, 3 (2007), 155--169.Google ScholarCross Ref
- Simon. 2007 b. Koli Calling comes of age: an analysis. In Seventh Baltic Sea Conference on Computing Education Research (Koli Calling 2007). 119--126. http://dl.acm.org/citation.cfm?id=2449323.2449336 Google ScholarDigital Library
- Simon, Judy Sheard, Angela Carbone, Michael de Raadt, Margaret Hamilton, Raymond Lister, and Errol Thompson. 2008. Eight years of computing education papers at NACCQ. In 21st Annual Conference of the National Advisory Committee on Computing Qualifications (NACCQ 2008). National Advisory Committee on Computing Qualifications, 101--107. https://www.citrenz.ac.nz/conferences/2008/101.pdfGoogle Scholar
- Beth Simon, Tzu-Yi Chen, Gary Lewandowski, Robert McCartney, and Kate Sanders. 2006. Commonsense computing: what students know before we teach (episode 1: sorting). In Second International Computing Education Research Workshop (ICER 2006). 29--40. Google ScholarDigital Library
- Teemu Sirki"a and Juha Sorva. 2015. How do students use program visualizations within an interactive ebook?. In 11th International Computing Education Research Conference (ICER 2015). 179--188. Google ScholarDigital Library
- Laura Toma and Jan Vahrenhold. 2018. Self-efficacy, cognitive load, and emotional reactions in collaborative algorithms labs -- a case study. In 14th International Computing Education Research Conference (ICER 2018). 1--10. Google ScholarDigital Library
- David W Valentine. 2004. CS educational research: a meta-analysis of SIGCSE Technical Symposium proceedings. In 35th ACM Technical Symposium on Computer Science Education (SIGCSE 2004). ACM, 255--259. Google ScholarDigital Library
- Jeffrey C Valentine and Harris Cooper. 2003. Effect size substantive interpretation guidelines: Issues in the interpretation of effect sizes. Washington, DC: What Works Clearinghouse (2003), 1--7.Google Scholar
Index Terms
- Inferential Statistics in Computing Education Research: A Methodological Review
Recommendations
Evolution of symposia on the interface of computing and statistics defines data science to be the interface
Goal of this article is to document evolution of the Interface and its Symposia, from their conception and birth when small data were analyzed with Statistics in the mid 20th Century until Big Data are now analyzed with Data Science in the early 21st ...
Graduate Level Research Methods and Statistics Courses: The Perspective of an Instructor
The goal of an educational system or degree program is to "educate" students. This immediately raises the question of what does it mean to "educate" students. All academic institutions, degree programs and content areas are typically expected to answer ...
Conducting Laboratory Experiments Properly with Statistical Tools: An Easy Hands-On Tutorial
WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data MiningThis hands-on half-day tutorial consists of two sessions. Part~I covers the following topics: Preliminaries; Paired and two-sample t-tests, confidence intervals; One-way ANOVA and two-way ANOVA without replication; Familiwise error rate. Part~II covers ...
Comments