Anjali Singh
ABSTRACT
Question generation as a form of learnersourcing is both a metacognitive learning activity for students that encourages the development of higher-order thinking skills and a method for producing question banks and assessments. To better understand the motivations for learners who engage in learnersourcing and its impacts on student learning, we conducted an experiment that measured the effects of Multiple Choice Question (MCQ) generation in an introductory data science MOOC. We compared two approaches to question generation: (i) as a required activity, and (ii) as an optional activity. In both cases, the learnersourcing activity was part of the student summative evaluation. We found that learners value creating questions more, and create higher quality questions when they choose to do so compared to when it is required. At the same time there is a significant reduction in instructor evaluation workload in large-scale courses when learners engage by choice due to self-selection. Thus, we propose choice-based learnersourcing as a new form of scalable personalized learning design for MOOCs in particular. In addition, we contribute an exploration of the factors that influence learner choice to create (or not create) an MCQ, which can help contextualize the propensity of learners to engage in such learnersourcing activities.
Supplemental Material
- Hervé Abdi. 2010. The Greenhouse-Geisser Correction. Encyclopedia of Research Design 1, 1 (2010), 544--548.Google Scholar
- Ester Aflalo. 2018. Students generating questions as a way of learning. Active Learning in Higher Education (2018), 1469787418769120.Google Scholar
- Andrew C Butler, Jeffrey D Karpicke, and Henry L Roediger III. 2007. The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied 13, 4 (2007), 273.Google ScholarCross Ref
- Christine Chin and David E Brown. 2002. Student-generated questions: A meaningful aspect of learning in science. International Journal of Science Education 24, 5 (2002), 521--549.Google ScholarCross Ref
- Christine Chin and Jonathan Osborne. 2008. Students' questions: a potential resource for teaching and learning science. Studies in science education 44, 1 (2008), 1--39.Google Scholar
- Ali Darvishi, Hassan Khosravi, and Shazia Sadiq. 2020. Utilising Learnersourcing to Inform Design Loop Adaptivity. In European Conference on Technology Enhanced Learning. Springer, 332--346.Google ScholarDigital Library
- Paul Denny, John Hamer, Andrew Luxton-Reilly, and Helen Purchase. 2008. PeerWise: students sharing their multiple choice questions. In Proceedings of the fourth international workshop on computing education research. 51--58.Google ScholarDigital Library
- Paul Denny, Brian Hanks, and Beth Simon. 2010. Peerwise: replication study of a student-collaborative self-testing web service in a us setting. In Proceedings of the 41st ACM technical symposium on Computer science education. 421--425.Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and John Hamer. 2008a. The PeerWise system of student contributed assessment questions. In Proceedings of the tenth conference on Australasian computing education-Volume 78. Citeseer, 69--74.Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and John Hamer. 2008b. Student use of the PeerWise system. In Proceedings of the 13th annual conference on Innovation and technology in computer science education. 73--77.Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2009. Quality of student contributed questions using PeerWise. In Proceedings of the Eleventh Australasian Conference on Computing Education-Volume 95. 55--63.Google ScholarDigital Library
- Paul Denny, Fiona McDonald, Ruth Empson, Philip Kelly, and Andrew Petersen. 2018. Empirical support for a causal relationship between gamification and learning outcomes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarDigital Library
- Yehudit J Dori and Orit Herscovitz. 1999. Question-posing capability as an alternative evaluation method: Analysis of an environmental case study. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching 36, 4 (1999), 411--430.Google ScholarCross Ref
- Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2623--2634.Google ScholarDigital Library
- Elaine Doyle, Patrick Buckley, and Joanne Whelan. 2019. Assessment co-creation: an exploratory analysis of opportunities and challenges based on student and instructor perspectives. Teaching in Higher Education 24, 6 (2019), 739--754.Google ScholarCross Ref
- Jacquelynne Eccles. 1983. Expectancies, values and academic behaviors. Achievement and achievement motives (1983).Google Scholar
- Thommy Eriksson, Tom Adawi, and Christian Stöhr. 2017. "Time is the bottleneck": a qualitative study exploring why learners drop out of MOOCs. Journal of Computing in Higher Education 29, 1 (2017), 133--146.Google ScholarCross Ref
- Alireza Farasat, Alexander Nikolaev, Suzanne Miller, and Rahul Gopalsamy. 2017. Crowdlearning: Towards collaborative problem-posing at scale. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. 221--224.Google ScholarDigital Library
- Elena L Glassman, Aaron Lin, Carrie J Cai, and Robert C Miller. 2016. Learnersourcing personalized hints. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 1626--1636.Google ScholarDigital Library
- Philip J Guo, Julia M Markel, and Xiong Zhang. 2020. Learnersourcing at Scale to Overcome Expert Blind Spots for Introductory Programming: A Three-Year Deployment Study on the Python Tutor Website. In Proceedings of the Seventh ACM Conference on Learning@ Scale. 301--304.Google ScholarDigital Library
- Judy Hardy, Simon P Bates, Morag M Casey, Kyle W Galloway, Ross K Galloway, Alison E Kay, Peter Kirsop, and Heather A McQueen. 2014. Student-generated content: Enhancing learning through sharing multiple-choice questions. International Journal of Science Education 36, 13 (2014), 2180--2194.Google ScholarCross Ref
- Maartje A Henderikx, Karel Kreijns, and Marco Kalz. 2017. Refining success and dropout in massive open online courses based on the intention--behavior gap. Distance Education 38, 3 (2017), 353--368.Google ScholarCross Ref
- Intrinsic Motivation Inventory. 1994. Intrinsic Motivation Inventory (IMI). The Intrinsic Motivation Inventory, Scale description (1994), 1--3.Google Scholar
- Hassan Khosravi, Gianluca Demartini, Shazia Sadiq, and Dragan Gavs ević. 2021. Charting the Design and Analytics Agenda of Learnersourcing Systems. In Proceedings of the 11th International Conference on Learning Analytics & Knowledge.Google ScholarDigital Library
- Juho Kim. 2014. Learnersourcing: Improving video learning with collective learner activity. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
- René F Kizilcec and Andrew J Saltarelli. 2019. Psychologically Inclusive Design. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, Vol. 10.Google ScholarDigital Library
- David R Krathwohl. 2002. A revision of Bloom's taxonomy: An overview. Theory into practice 41, 4 (2002), 212--218.Google Scholar
- Shang-Wen Daniel Li and Piotr Mitros. 2015. Learnersourced recommendations for remediation. In 2015 IEEE 15th International Conference on Advanced Learning Technologies. IEEE, 411--412.Google Scholar
- Andrew Luxton-Reilly, Daniel Bertinshaw, Paul Denny, Beryl Plimmer, and Robert Sheehan. 2012. The impact of question generation activities on performance. In Proceedings of the 43rd ACM technical symposium on Computer Science Education. 391--396.Google ScholarDigital Library
- Mitchell J Nathan, Kenneth R Koedinger, Martha W Alibali, and others. 2001. Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In Proceedings of the third international conference on cognitive science, Vol. 644648.Google Scholar
- Helen Purchase, John Hamer, Paul Denny, and Andrew Luxton-Reilly. 2010. The quality of a PeerWise MCQ repository. In Proceedings of the Twelfth Australasian Conference on Computing Education-Volume 103. Citeseer, 137--146.Google ScholarDigital Library
- Amy Rechkemmer and Ming Yin. 2020. Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 122--131.Google ScholarCross Ref
- Falko Rheinberg, Regina Vollmeyer, and Wolfram Rollett. 2000. Motivation and action in self-regulated learning. In Handbook of self-regulation. Elsevier, 503--529.Google ScholarCross Ref
- Dan Rothstein and Luz Santana. 2011. Make just one change: Teach students to ask their own questions. Harvard Education Press.Google Scholar
- Richard Michael Ryan and Edward Lewis Deci. 2016. Facilitating and hindering motivation, learning, and well-being in schools: Research and observations from self-determination theory. Handbook of motivation at school 96 (2016).Google Scholar
- Amanda Sykes, Paul Denny, and Lesley Nicolson. 2011. PeerWise-the marmite of veterinary student learning. In Proceedings of the 10th European Conference on e-Learning Brighton Business School, University of Brighton, UK. 10--11 November, 2011, Vols 1 and 2 (S Greener, A Rospigliosi, eds.),. Academic Conferences Ltd, 820--830.Google Scholar
- Peter Van Rosmalen, Julia Kasch, Marco Kalz, Olga Firssova, and Francis Brouns. 2017. Towards "MOOCs with a Purpose': Crowdsourcing and analysing scalable design solutions with MOOC learners. In European Conference on Technology Enhanced Learning. Springer, 486--491.Google ScholarCross Ref
- Xu Wang, Srinivasa Teja Talluri, Carolyn Rose, and Kenneth Koedinger. 2019. UpGrade: Sourcing Student Open-Ended Solutions to Create Scalable Learning Opportunities. In Proceedings of the Sixth (2019) ACM Conference on Learning@ Scale. 1--10.Google ScholarDigital Library
- Allan Wigfield. 1994. Expectancy-value theory of achievement motivation: A developmental perspective. Educational psychology review 6, 1 (1994), 49--78.Google Scholar
- Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. 379--388.Google ScholarDigital Library
- Fu-Yun Yu. 2012. Learner-centered pedagogy+adaptable and scaffolded learning space design-online student question-generation.. In International Conference on Computers in Education 2012.Google Scholar
- Haichao Zheng, Dahui Li, and Wenhua Hou. 2011. Task design, motivation, and participation in crowdsourcing contests. International Journal of Electronic Commerce 15, 4 (2011), 57--88.Google ScholarDigital Library
- Saijing Zheng, Mary Beth Rosson, Patrick C Shih, and John M Carroll. 2015. Understanding student motivation, behaviors and perceptions in MOOCs. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 1882--1895.Google ScholarDigital Library
Index Terms
- What's In It for the Learners? Evidence from a Randomized Field Experiment on Learnersourcing Questions in a MOOC
Recommendations
Perception of MOOC Pedagogical Tools and Learners' Learning Styles in MOOC Blended Teaching: a Case Study
ICEBT '19: Proceedings of the 2019 3rd International Conference on E-Education, E-Business and E-TechnologyRapid development has been achieved since the emergence of MOOC in 2008, but there are still many defects in the popularization of MOOC. Developing blended teaching by utilizing is considered to be one of effective means to overcome these shortcomings. ...
Self-Regulation for High School Learners in a MOOC Computer Science Course
SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science EducationCourses designed for Massive Open Online Courseware (MOOC)platforms provide learners worldwide with extensive learning opportunities. Previous research has explored learner motivation in MOOC courses using self-regulated learning (SRL) theory. How-ever; ...
Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing
LAK22: LAK22: 12th International Learning Analytics and Knowledge ConferenceThe benefits of incorporating scaffolds that promote strategies of self-regulated learning (SRL) to help student learning are widely studied and recognised in the literature. However, the best methods for incorporating them in educational technologies ...
Comments