skip to main content
10.1145/3341525.3387375acmconferencesArticle/Chapter ViewAbstractPublication PagesiticseConference Proceedingsconference-collections
research-article

Depth of Use: An Empirical Framework to Help Faculty Gauge the Relative Impact of Learning Management System Tools

Published: 15 June 2020 Publication History

Abstract

Learning management system (LMS) tools are increasingly relevant to scaling computing pedagogies. Measuring their utilization and impact at scale, however, remains computationally expensive. We examine the problem of estimating the utilization of a department-wide LMS, and its impact on the design, management and outcomes of Computer Science courses. We introduce 'depth-of-use' (DOU): a first-principles, resource-specific metric of LMS utilization. We then hypothesis-test the relationship between DOU and course attributes like modality (course level, mode-of-delivery, third-party app use), participation (enrollment, viewership), logistics (teaching support, digital skills training) and outcomes (average GPA, DFW rate). Experiments on metadata from over 1300 Computer Science courses taught at Virginia Tech between 2015 and 2019 suggest that our framing of DOU helps identify resource-level preferences of micro-cohorts of courses, linked to their content, logistics and pedagogies. We discover that, across the Computer Science department at Virginia Tech, overall LMS use is consistently linked to favorable learning outcomes. We also discover that a complex interaction between the needs for scale, ubiquitous access and interoperability drives strong LMS utilization, with graduate and online-only courses faring highest in their aggregate use of LMS services. Finally, we describe two key applications of our analyses. One, we demonstrate how DOU can help CS faculty identify the relative impact of transition from legacy apps to LMS services. Two, we describe how DOU can help instructional designers evaluate and improve their design interventions.

References

[1]
Tella Adeyinka. 2011. Reliability and factor analysis of a blackboard course management system success: A scale development and validation in an educational context. Journal of Information Technology Education: Research, Vol. 10 (2011), 55--80.
[2]
Tella Adeyinka and Stephen Mutula. 2010. A proposed model for evaluating the success of WebCT course content management system. Computers in Human Behavior, Vol. 26, 6 (2010), 1795--1805.
[3]
Anders Berggren, Daniel Burgos, Josep M Fontana, Don Hinkelman, Vu Hung, Anthony Hursh, and Ger Tielemans. 2005. Practical and Pedagogical Issues for Teacher Adoption of IMS Learning Design Standards in Moodle LMS. Journal of Interactive Media in Education, Vol. 2, 02 (2005), 2.
[4]
Hamish Coates, Richard James, and Gabrielle Baldwin. 2005. A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, Vol. 11 (2005), 19--36.
[5]
William H DeLone and Ephraim R McLean. 1992. Information systems success: The quest for the dependent variable. Information systems research, Vol. 3, 1 (1992), 60--95.
[6]
William H Delone and Ephraim R McLean. 2003. The DeLone and McLean model of information systems success: a ten-year update. Journal of management information systems, Vol. 19, 4 (2003), 9--30.
[7]
edutechnica. 2018. 6th Annual LMS Data Update. https://edutechnica.com/2018/10/06/6th-annual-lms-data-update/
[8]
Taha Hassan. 2019. On bias in social reviews of university courses. In Companion Publication of the 10th ACM Conference on Web Science. 11--14.
[9]
Taha Hassan and D. Scott McCrickard. 2019. Trust and trustworthiness in social recommender systems. In Companion Proceedings of the 2019 World Wide Web Conference (WWW '19 Companion). ACM.
[10]
William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, Vol. 47, 260 (1952), 583--621.
[11]
Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50--60.
[12]
Tanya J McGill and Jane E Klobas. 2009. A task--technology fit view of learning management system impact. Computers & Education, Vol. 52, 2 (2009), 496--508.
[13]
Joel Mtebe. 2015. Learning management system success: Increasing learning management system usage in higher education in sub-Saharan Africa. International Journal of Education and Development using ICT, Vol. 11, 2 (2015).
[14]
Eric WT Ngai, JKL Poon, and YHC Chan. 2007. Empirical examination of the adoption of WebCT using TAM. Computers & education, Vol. 48, 2 (2007), 250--267.
[15]
Sevgi Ozkan and Refika Koseler. 2009. Multi-dimensional students' evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, Vol. 53, 4 (2009), 1285--1296.
[16]
Prajish Prasad. 2018. Developing Students' Cognitive Processes Required for Software Design Verification. In Proceedings of the 2018 ACM Conference on International Computing Education Research. ACM, 284--285.
[17]
Guido Rößling, Mike Joy, Andrés Moreno, Atanas Radenski, Lauri Malmi, Andreas Kerren, Thomas Naps, Rockford J Ross, Michael Clancy, Ari Korhonen, et al. 2008. Enhancing learning management systems to better support computer science education. ACM SIGCSE Bulletin, Vol. 40, 4 (2008), 142--166.
[18]
Guido Rößling, Myles McNally, Pierluigi Crescenzi, Atanas Radenski, Petri Ihantola, and M Gloria Sánchez-Torrubia. 2010. Adapting moodle to better support CS education. In Proceedings of the 2010 ITiCSE working group reports. ACM, 15--27.
[19]
Virginia Tech, TLOS. 2019. Upcoming Offerings, Networked Learning Initiatives. https://app.nli.tlos.vt.edu/
[20]
Bryan R Warnick. 2009. Imitation and education: A philosophical inquiry into learning by example .SUNY Press.
[21]
Richard E West, Greg Waddoups, and Charles R Graham. 2007. Understanding the experiences of instructors as they adopt a course management system. Educational Technology Research and Development, Vol. 55, 1 (2007), 1--26.
[22]
Diane Wilcox, Jane Thall, and Oris Griffin. 2016. One canvas, two audiences: How faculty and students use a newly adopted learning management system. In Society for Information Technology & Teacher Education International Conference. Association for the Advancement of Computing in Education (AACE), 1163--1168.

Cited By

View all
  • (2024)Simplify, Consolidate, Intervene: Facilitating Institutional Support with Mental Models of Learning Management System UseProceedings of the ACM on Human-Computer Interaction10.1145/36870518:CSCW2(1-23)Online publication date: 8-Nov-2024
  • (2023)Data-Informed Learning Design in a Computer Science CourseLearning, Design, and Technology10.1007/978-3-319-17461-7_176(2323-2345)Online publication date: 15-Oct-2023
  • (2022)Data-Informed Learning Design in a Computer Science CourseLearning, Design, and Technology10.1007/978-3-319-17727-4_176-1(1-23)Online publication date: 18-Aug-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education
June 2020
615 pages
ISBN:9781450368742
DOI:10.1145/3341525
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adoption
  2. learning management system
  3. utilization

Qualifiers

  • Research-article

Conference

ITiCSE '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Upcoming Conference

ITiCSE '25
Innovation and Technology in Computer Science Education
June 27 - July 2, 2025
Nijmegen , Netherlands

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)4
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Simplify, Consolidate, Intervene: Facilitating Institutional Support with Mental Models of Learning Management System UseProceedings of the ACM on Human-Computer Interaction10.1145/36870518:CSCW2(1-23)Online publication date: 8-Nov-2024
  • (2023)Data-Informed Learning Design in a Computer Science CourseLearning, Design, and Technology10.1007/978-3-319-17461-7_176(2323-2345)Online publication date: 15-Oct-2023
  • (2022)Data-Informed Learning Design in a Computer Science CourseLearning, Design, and Technology10.1007/978-3-319-17727-4_176-1(1-23)Online publication date: 18-Aug-2022
  • (2021)Learning to Trust: Understanding Editorial Authority and Trust in Recommender Systems for EducationProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3450613.3456811(24-32)Online publication date: 21-Jun-2021
  • (2021)Evaluation of student collaboration on canvas LMS using educational data mining techniquesProceedings of the 2021 ACM Southeast Conference10.1145/3409334.3452042(55-62)Online publication date: 15-Apr-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media