research-article

Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC

Authors:

Joseph A. Konstan,

J. D. Walker,

D. Christopher Brooks,

Keith Brown,

Michael D. EkstrandAuthors Info & Claims

ACM Transactions on Computer-Human Interaction (TOCHI), Volume 22, Issue 2

Article No.: 10, Pages 1 - 23

https://doi.org/10.1145/2728171

Published: 15 April 2015 Publication History

Get Access

Abstract

In the fall of 2013, we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of offering this course was to experiment with this type of instruction, we performed extensive evaluation including surveys of demographics, self-assessed skills, and learning intent; we also designed a knowledge-assessment tool specifically for the subject matter in this course, administering it before and after the course to measure learning, and again 5 months later to measure retention. We also tracked students through the course, including separating out students enrolled for credit from those enrolled only for the free, open course.

Students had significant knowledge gains across all levels of prior knowledge and across all demographic categories. The main predictor of knowledge gain was effort expended in the course. Students also had significant knowledge retention after the course. Both of these results are limited to the sample of students who chose to complete our knowledge tests. Student completion of the course was hard to predict, with few factors contributing predictive power; the main predictor of completion was intent to complete. Students who chose a concepts-only track with hand exercises achieved the same level of knowledge of recommender systems concepts as those who chose a programming track and its added assignments, though the programming students gained additional programming knowledge. Based on the limited data we were able to gather, face-to-face students performed as well as the online-only students or better; they preferred this format to traditional lecture for reasons ranging from pure convenience to the desire to watch videos at a different pace (slower for English language learners; faster for some native English speakers). This article also includes our qualitative observations, lessons learned, and future directions.

References

[1]

John P. Buerck, Srikanth P. Mudigonda, Stephanie E. Mooshegian, Kyle Collins, Nicholas Grimm, Kristen Bonney, and Hadley Kombrink. 2013. Predicting non-traditional student learning outcomes using data analytics--a pilot research study. Journal of the Computer Science College 28, 5 (May 2013), 260--265.

Digital Library

Google Scholar

[2]

J. Champaign, K. F. Colvin, A. Liu, C. Fredericks, D. Seaton, and D. E. Pritchard. 2014. Correlating skill and improvement in 2 MOOCs with a student's time on tasks. Proceedings of the 1st ACM Conference on Learning @ Scale. ACM, New York, 11--20.

Digital Library

Google Scholar

[3]

Doug Clow. 2013. MOOCs and the funnel of participation. In Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK’13). ACM, New York, NY, 185--189.

Digital Library

Google Scholar

[4]

K. Colvin, J. Champaign, A. Liu, Q. Zhou, C. Fredericks, and D. Pritchard. 2014. Learning in an introductory physics MOOC: All cohorts learn equally, including an on-campus class. International Review of Research in Open and Distance Learning 15, 4.

Crossref

Google Scholar

[5]

J. DeBoer, A. D. Ho, G. S. Stump, and L. Breslow. 2014. Changing “course”: Reconceptualizing educational variables for massive open online courses. Educational Researcher 43, 2, 74--84.

Crossref

Google Scholar

[6]

M. D. Ekstrand, M. Ludwig, J. A. Konstan, and J. T. Riedl. 2011. Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys’11). ACM, New York, NY, 133--140.

Digital Library

Google Scholar

[7]

E. J. Emanuel. 2013. Online education: MOOCs taken by educated few. Nature 503, 7476, 342--342.

Google Scholar

[8]

René F. Kizilcec, Chris Piech, and Emily Schneider. 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the 3rd International Conference on Learning Analytics and Knowledge.

Digital Library

Google Scholar

[9]

J. A. Konstan, J. D. Walker, D. C. Brooks, K. Brown, and M. D. Ekstrand. 2014. Teaching recommender systems at large scale: Evaluation and lessons learned from a hybrid MOOC. In Proceedings of the 1st ACM Conference on Learning@Scale. 61--70.

Digital Library

Google Scholar

[10]

D. A. McConnell, D. N. Steer, K. D. Owens, C. Knight. 2005. How students think: Implications for learning in introductory geoscience courses. Journal of Geoscience Education, 53, 4, 462--470.

Crossref

Google Scholar

[11]

B. Means, Y. Toyama, R. Murphy, M. Bakia, and K. Jones. 2010. Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Center for Technology in Learning, U.S. Department of Education, Washington, DC.

Google Scholar

[12]

E. T. Pascarella and P. T. Terenzini. 2005. How College Affects Students: Volume 2, A Third Decade of Research. Jossey-Bass, San Francisco, CA.

Google Scholar

[13]

G. B. Semb, J. A. Ellis, and J. Araujo. 1993. Long-term memory for knowledge learned in school. Journal of Educational Psychology 85, 2, 305--316.

Crossref

Google Scholar

[14]

Julia Wilkowski, Amit Deutsch, and Daniel M. Russell. 2014. Student skill and goal achievement in the mapping with google MOOC. In Proceedings of the 1st ACM conference on Learning @ Scale Conference (L@S’14). ACM, New York, NY, 3--10.

Digital Library

Google Scholar

Cited By

View all

Li YZhao M(2024)The study on the influence factors of intention to continue using MOOCs: integrating UTAUT model and social presenceInteractive Learning Environments10.1080/10494820.2024.231856232:10(7431-7449)Online publication date: 23-Feb-2024
https://doi.org/10.1080/10494820.2024.2318562
Xia XQi W(2024)Multilayer knowledge graph construction and learning behavior routing guidance based on implicit relationships of MOOCsTechnological Forecasting and Social Change10.1016/j.techfore.2024.123442204(123442)Online publication date: Jul-2024
https://doi.org/10.1016/j.techfore.2024.123442
Rulinawaty Purwanto ASamboteng LKasmad MBasit M(2023)Global Trends and Policy Strategies and Their Implications for the Sustainable Development of MOOCs in IndonesiaProceedings of the Fourth International Conference on Administrative Science (ICAS 2022)10.2991/978-2-38476-104-3_47(491-508)Online publication date: 29-Sep-2023
https://doi.org/10.2991/978-2-38476-104-3_47
Show More Cited By

Index Terms

Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC
1. Applied computing
  1. Education
    1. Distance learning
    2. E-learning
2. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

Teaching recommender systems at large scale: evaluation and lessons learned from a hybrid MOOC
L@S '14: Proceedings of the first ACM conference on Learning @ scale conference

In Fall 2013 we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of ...
A case study of learning assessment, flow, satisfaction and expressiveness of the music game: focused on elementary students' music appreciation learning

This study examines the effectiveness of a music game for music appreciation education on learners' learning assessment, learning flow, satisfaction, and expressiveness of elementary students. For the experiment, fourth grade students were divided into ...
Dual Teaching: Simultaneous Remote and In-Person Learning During COVID
SIGITE '21: Proceedings of the 22nd Annual Conference on Information Technology Education

Social distancing guidelines put in place to combat COVID-19 resulted in a general education introductory information technology course being taught in a dual teaching environment. Each lesson, some students attended in-person while simultaneously ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 22, Issue 2

Special Issue on Online Learning at Scale

April 2015

133 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/2744768

Editor:
Shumin Zhai
Google, Inc.

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2015

Accepted: 01 January 2015

Revised: 01 January 2015

Received: 01 June 2014

Published in TOCHI Volume 22, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
1,646
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)6

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Li YZhao M(2024)The study on the influence factors of intention to continue using MOOCs: integrating UTAUT model and social presenceInteractive Learning Environments10.1080/10494820.2024.231856232:10(7431-7449)Online publication date: 23-Feb-2024
https://doi.org/10.1080/10494820.2024.2318562
Xia XQi W(2024)Multilayer knowledge graph construction and learning behavior routing guidance based on implicit relationships of MOOCsTechnological Forecasting and Social Change10.1016/j.techfore.2024.123442204(123442)Online publication date: Jul-2024
https://doi.org/10.1016/j.techfore.2024.123442
Rulinawaty Purwanto ASamboteng LKasmad MBasit M(2023)Global Trends and Policy Strategies and Their Implications for the Sustainable Development of MOOCs in IndonesiaProceedings of the Fourth International Conference on Administrative Science (ICAS 2022)10.2991/978-2-38476-104-3_47(491-508)Online publication date: 29-Sep-2023
https://doi.org/10.2991/978-2-38476-104-3_47
Zhang Y(2023)Research on the innovation of ideological and political education theory and practice in colleges and universities under the background of big dataApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.2.001179:1Online publication date: 1-Aug-2023
https://doi.org/10.2478/amns.2023.2.00117
Murad DToha MMayatopani HDewi MLeandros RMurad SHeryadi YWijanarko B(2023)The Impact of Recommendation System Provided Based on Online Learning Readiness in the MOOC Environment2023 24th International Arab Conference on Information Technology (ACIT)10.1109/ACIT58888.2023.10453910(1-6)Online publication date: 6-Dec-2023
https://doi.org/10.1109/ACIT58888.2023.10453910
Yang XTan L(2022)The Construction of Accurate Recommendation Model of Learning Resources of Knowledge Graph under Deep LearningScientific Programming10.1155/2022/10101222022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/1010122
Lau SEldridge JEllis SFraenkel ALanglois MRampure STiefenbruck JGuo PKizilcec RDavis KOchoa X(2022)The Challenges of Evolving Technical Courses at Scale: Four Case Studies of Updating Large Data Science CoursesProceedings of the Ninth ACM Conference on Learning @ Scale10.1145/3491140.3528278(201-211)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1145/3491140.3528278
Li YZhao M(2021)A Study on the Influencing Factors of Continued Intention to Use MOOCs: UTAUT Model and CCC Moderating EffectFrontiers in Psychology10.3389/fpsyg.2021.52825912Online publication date: 4-Aug-2021
https://doi.org/10.3389/fpsyg.2021.528259
Bai XZhang FLi JGuo TAziz AJin AXia F(2021)Educational Big Data: Predictions, Applications and ChallengesBig Data Research10.1016/j.bdr.2021.100270(100270)Online publication date: Sep-2021
https://doi.org/10.1016/j.bdr.2021.100270
Ekstrand Md'Aquin MDietze SHauff CCurry ECudre Mauroux P(2020)LensKit for Python: Next-Generation Software for Recommender Systems ExperimentsProceedings of the 29th ACM International Conference on Information & Knowledge Management10.1145/3340531.3412778(2999-3006)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3340531.3412778
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations