research-article

The Challenges of Evolving Technical Courses at Scale: Four Case Studies of Updating Large Data Science Courses

Authors:

Justin Eldridge,

Aaron Fraenkel,

Marina Langlois,

Janine Tiefenbruck,

Philip J. GuoAuthors Info & Claims

L@S '22: Proceedings of the Ninth ACM Conference on Learning @ Scale

Pages 201 - 211

https://doi.org/10.1145/3491140.3528278

Published: 01 June 2022 Publication History

Abstract

Instructors who teach large-scale technical courses, especially on data science and programming, must do a large amount of logistical work when updating their courses. All of this behind-the-scenes labor takes time away from the pedagogically-meaningful work of teaching students. Over the past five years, the authors of this paper have created and updated eight courses for an undergraduate data science program that serves over 2,000 students per year. We present four case studies from our teaching experiences that highlight major challenges in maintaining and updating technical courses: 1) There were intricate dependencies between course materials, so making updates to one part of the course would require updating many other parts. 2) We needed to maintain several variants of course materials such as assignments. 3) We wrote large amounts of ad-hoc custom software infrastructure to manage logistics. 4) We could not easily reuse software written by others. Our case studies point to design ideas for instructor-oriented tools that can reduce the logistical complexities of teaching at scale, thus letting instructors focus on the substance of teaching rather than on mundane logistics.

References

[1]

2018. What's New In Python 3.7 - Python 3.9.6 Documentation. https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-37.

[2]

2021. Project Gutenberg. https://www.gutenberg.org/.

[3]

2022. Otter-Grader Documentation - Otter-Grader Documentation. https://otter-grader.readthedocs.io/en/latest/.

[4]

Computing Research Association. 2017. Generation CS: Computer Science Undergraduate Enrollments Surge Since 2006. https://cra.org/data/generation-cs/. Accessed: 2021-01--15.

[5]

Soumya Basu, Albert Wu, Brian Hou, and John DeNero. 2015. Problems Before Solutions: Automated Problem Clarification at Scale. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S '15). Association for Computing Machinery, New York, NY, USA, 205--213. https://doi.org/10.1145/2724660.2724679

Digital Library

[6]

Ned Chapin, Joanne E. Hale, Khaled Md. Kham, Juan F. Ramil, and Wui-Gee Tan. 2001. Types of Software Evolution and Software Maintenance. Journal of Software Maintenance 13, 1 (Jan. 2001), 3--30.

[7]

Hao-Fei Cheng, Bowen Yu, Siwei Fu, Jian Zhao, Brent Hecht, Joseph Konstan, Loren Terveen, Svetlana Yarosh, and Haiyi Zhu. 2019. Teaching UI Design at Global Scales: A Case Study of the Design of Collaborative Capstone Projects for MOOCs. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale (Chicago, IL, USA) (L@S '19). Association for Computing Machinery, New York, NY, USA, Article 11, 11 pages. https://doi.org/10.1145/3330430.3333635

Digital Library

[8]

Anna T. Cianciolo and Robert J. Sternberg. 2018. Practical Intelligence and Tacit Knowledge: An Ecological View of Expertise (2 ed.). Cambridge University Press, 770--792. https://doi.org/10.1017/9781316480748.039

[9]

Russ Cox. 2019. Surviving Software Dependencies. Commun. ACM 62, 9 (Aug. 2019), 36--43. https://doi.org/10.1145/3347446

Digital Library

[10]

Laura Dabbish, Colleen Stuart, Jason Tsay, and Jim Herbsleb. 2012. Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW '12). Association for Computing Machinery, New York, NY, USA, 1277--1286. https://doi.org/10.1145/2145204.2145396

Digital Library

[11]

John DeNero. 2021. Data 8. http://data8.org/.

[12]

Nadia Eghbal. 2020. Working in Public: The Making and Maintenance of Open Source Software. Stripe Press.

[13]

Stuart I. Feldman. 1979. Make--A program for maintaining computer programs. Software: Practice and experience 9, 4 (1979), 255--265.

[14]

Joseph Feliciano, Margaret-Anne Storey, and Alexey Zagalsky. 2016. Student Experiences Using GitHub in Software Engineering Courses: A Case Study. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). 422--431.

Digital Library

[15]

R. Stuart Geiger, Nelle Varoquaux, Charlotte Mazel-Cabasse, and Chris Holdgraf. 2018. The types, roles, and practices of documentation in data analytics open source software libraries. Computer Supported Cooperative Work (CSCW) 27, 3 (2018), 767--802.

Digital Library

[16]

Michael W. Godfrey and Daniel M. German. 2008. The past, present, and future of software evolution. In 2008 Frontiers of Software Maintenance. 129--138. https://doi.org/10.1109/FOSM.2008.4659256

[17]

Aaron Halfaker, R. Stuart Geiger, Jonathan T. Morgan, and John Riedl. 2013. The Rise and Decline of an Open Collaboration System: How Wikipedia's Reaction to Popularity Is Causing Its Decline. American Behavioral Scientist 57, 5 (2013), 664--688. https://doi.org/10.1177/0002764212469365 arXiv:https://doi.org/10.1177/0002764212469365

[18]

David A. Joyner and Charles Isbell. 2019. Master's at Scale: Five Years in a Scalable Online Graduate Degree. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale (Chicago, IL, USA) (L@S '19). Association for Computing Machinery, New York, NY, USA, Article 21, 10 pages. https://doi.org/10.1145/3330430.3333630

Digital Library

[19]

Alexander C. Kafka. 2018. With Student Interest Soaring, Berkeley Creates New Data-Sciences Division. The Chronicle of Higher Education (Nov 2018).

[20]

Aniket Kittur and Robert E. Kraut. 2010. Beyond Wikipedia: Coordination and Conflict in Online Production Groups. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (Savannah, Georgia, USA) (CSCW '10). Association for Computing Machinery, New York, NY, USA, 215--224. https://doi.org/10.1145/1718918.1718959

Digital Library

[21]

Carlos Delgado Kloos, Ma Blanca Ibáñez, Carlos Alario-Hoyos, Pedro J Munoz-Merino, Iria Estévez Ayres, Carmen Fernández Panadero, and Julio Villena. 2016. From software engineering to courseware engineering. In 2016 IEEE Global Engineering Education Conference (EDUCON). IEEE, 1122--1128.

[22]

Joseph A. Konstan, J. D. Walker, D. Christopher Brooks, Keith Brown, and Michael D. Ekstrand. 2015. Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC. ACM Trans. Comput.-Hum. Interact. 22, 2, Article 10 (apr 2015), 23 pages. https://doi.org/10.1145/2728171

Digital Library

[23]

Sean Kross and Philip J. Guo. 2019. Practitioners Teaching Data Science in Industry and Academia: Expectations, Workflows, and Challenges. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). ACM, New York, NY, USA, Article 263, 14 pages. https://doi.org/10.1145/3290605.3300493

Digital Library

[24]

Philippe Kruchten, Robert L. Nord, and Ipek Ozkaya. 2012. Technical Debt: From Metaphor to Theory and Practice. IEEE Software 29, 6 (2012), 18--21. https://doi.org/10.1109/MS.2012.167

Digital Library

[25]

Chinmay Kulkarni, Julia Cambre, Yasmine Kotturi, Michael S. Bernstein, and Scott R. Klemmer. 2015. Talkabout: Making Distance Matter with Small Groups in Massive Classes. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). Association for Computing Machinery, New York, NY, USA, 1116--1128. https://doi.org/10.1145/2675133.2675166

Digital Library

[26]

Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S '15). Association for Computing Machinery, New York, NY, USA, 75--84. https://doi.org/10.1145/2724660.2724670

Digital Library

[27]

Sam Lau, Ian Drosos, Julia M. Markel, and Philip J. Guo. 2020. The Design Space of Computational Notebooks: An Analysis of 60 Systems in Academia and Industry. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (VL/HCC '20).

[28]

David Malan. 2021. CS50 Docs: ALL THE DOCS! https://cs50.readthedocs.io/. Accessed: 2021-01--15.

[29]

David J. Malan. 2013. CS50 Sandbox: Secure Execution of Untrusted Code. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE '13). Association for Computing Machinery, New York, NY, USA, 141--146. https://doi.org/10.1145/2445196.2445242

Digital Library

[30]

Wes McKinney. 2011. Pandas: A Foundational Python Library for Data Analysis and Statistics. Python for high performance and scientific computing 14, 9 (2011).

[31]

Diba Mirza, Phillip T. Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate Teaching Assistants in Computer Science: A Systematic Literature Review. In Proceedings of the 2019 ACM Conference on International Computing Education Research (Toronto ON, Canada) (ICER '19). Association for Computing Machinery, New York, NY, USA, 31--40. https://doi.org/10.1145/3291279.3339422

Digital Library

[32]

Alok Mysore and Philip J. Guo. 2017. Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). ACM, New York, NY, USA, 703--714. https://doi.org/10.1145/3126594.3126628

Digital Library

[33]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. 2011. Scikit-Learn: Machine Learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830.

[34]

Jeffrey M. Perkel. 2018. Why Jupyter Is Data Scientists' Computational Notebook of Choice. Nature 563, 7732 (2018), 145--147.

[35]

Chris Piech, Lisa Yan, Lisa Einstein, Ana Saavedra, Baris Bozkurt, Eliska Sestakova, Ondrej Guth, and Nick McKeown. 2020. Co-Teaching Computer Science Across Borders: Human-Centric Learning at Scale. In Proceedings of the Seventh ACM Conference on Learning @ Scale (Virtual Event, USA) (L@S '20). Association for Computing Machinery, New York, NY, USA, 103--113. https://doi.org/10.1145/3386527.3405915

Digital Library

[36]

Chad Sharp, Jelle van Assema, Brian Yu, Kareem Zidane, and David J. Malan. 2020. An Open-Source, API-Based Framework for Assessing the Correctness of Code in CS50. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE '20). Association for Computing Machinery, New York, NY, USA, 487--492. https://doi.org/10.1145/3341525.3387417

Digital Library

[37]

Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017. Gradescope: A Fast, Flexible, and Fair System for Scalable Assessment of Handwritten Work. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge, Massachusetts, USA) (L@S '17). Association for Computing Machinery, New York, NY, USA, 81--88. https://doi.org/10.1145/3051457.3051466

Digital Library

[38]

Sumukh Sridhara, Brian Hou, Jeffrey Lu, and John DeNero. 2016. Fuzz Testing Projects in Massive Courses. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (Edinburgh, Scotland, UK) (L@S '16). Association for Computing Machinery, New York, NY, USA, 361--367. https://doi.org/10.1145/2876034.2876050

Digital Library

[39]

Beckie Supiano. 2018. It Matters a Lot Who Teaches Introductory Courses. Here's Why. The Chronicle of Higher Education (Apr 2018).

[40]

Wikipedia. 2021. Package manager. https://en.wikipedia.org/wiki/Package_manager. Accessed: 2021-01--15.

[41]

Wikipedia. 2021. Pivot table. https://en.wikipedia.org/wiki/Pivot_table. Accessed: 2021-01--15.

[42]

Alexey Zagalsky, Joseph Feliciano, Margaret-Anne Storey, Yiyun Zhao, and Weiliang Wang. 2015. The Emergence of GitHub as a Collaborative Platform for Education. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). Association for Computing Machinery, New York, NY, USA, 1906--1917. https://doi.org/10.1145/2675133.2675284

Digital Library

Cited By

Savonen CWright CHoffman AHumphries ECox KTan FLeek J(2024)Motivation, inclusivity, and realism should drive data science educationF1000Research10.12688/f1000research.134655.212(1240)Online publication date: 4-Apr-2024
https://doi.org/10.12688/f1000research.134655.2
Choe JLee MMarathe SFox AGarcia DNorouzi NStephenson BStone JBattestilli LRebelsky SShoop L(2024)Data Science Mastery Learning Using Parsons Problems-Inspired Table TransformationsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635556(1604-1605)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635556
Savonen CWright CHoffman AHumphries ECox KTan FLeek J(2023)Motivation, inclusivity, and realism should drive data science educationF1000Research10.12688/f1000research.134655.112(1240)Online publication date: 28-Sep-2023
https://doi.org/10.12688/f1000research.134655.1
Show More Cited By

Index Terms

The Challenges of Evolving Technical Courses at Scale: Four Case Studies of Updating Large Data Science Courses
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

Bringing security into traditional computer science courses: challenges and support (abstract only)
SIGCSE '13: Proceeding of the 44th ACM technical symposium on Computer science education

In today's world, it is critical that our students get exposure to security education as part of their Computer Science (CS) curriculum. The lack of security consciousness in digital workforce is costing businesses, government, and citizens severely ...
Case Studies of Use: Creating Counselor Champions for Change in K12 Computing Education (Abstract Only)
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

Counselors for Computing leverages the National Center for Women & Information Technology (NCWIT)'s organizational membership, applies evidence-based approaches for professional development, and monitors and adjusts practices to make inroads into CS ...
Computational thinking in high school courses
SIGCSE '10: Proceedings of the 41st ACM technical symposium on Computer science education

The number of undergraduates entering computer science has declined in recent years. This is paralleled by a drop in the number of high school students taking the CS AP exam and the number of high schools offering computer science courses. The declines ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

L@S '22: Proceedings of the Ninth ACM Conference on Learning @ Scale

June 2022

491 pages

ISBN:9781450391580

DOI:10.1145/3491140

General Chair:
René F. Kizilcec
Cornell University, USA
,
Program Chairs:
Katie Davis
University of Washington, USA
,
Xavier Ochoa
New York University, USA

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2022

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

L@S '22

L@S '22: Ninth (2022) ACM Conference on Learning @ Scale

June 1 - 3, 2022

NY, New York City, USA

Acceptance Rates

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
298
Total Downloads

Downloads (Last 12 months)147
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Savonen CWright CHoffman AHumphries ECox KTan FLeek J(2024)Motivation, inclusivity, and realism should drive data science educationF1000Research10.12688/f1000research.134655.212(1240)Online publication date: 4-Apr-2024
https://doi.org/10.12688/f1000research.134655.2
Choe JLee MMarathe SFox AGarcia DNorouzi NStephenson BStone JBattestilli LRebelsky SShoop L(2024)Data Science Mastery Learning Using Parsons Problems-Inspired Table TransformationsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635556(1604-1605)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635556
Savonen CWright CHoffman AHumphries ECox KTan FLeek J(2023)Motivation, inclusivity, and realism should drive data science educationF1000Research10.12688/f1000research.134655.112(1240)Online publication date: 28-Sep-2023
https://doi.org/10.12688/f1000research.134655.1
Mirhosseini SHenley AParnin CDoyle MStephenson BDorn BSoh LBattestilli L(2023)What Is Your Biggest Pain Point?Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569816(291-297)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569816

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten