skip to main content
10.1145/3231644.3231669acmotherconferencesArticle/Chapter ViewAbstractPublication Pagesl-at-sConference Proceedingsconference-collections
research-article

OARS: exploring instructor analytics for online learning

Published: 26 June 2018 Publication History

Abstract

Learning analytics systems have the potential to bring enormous value to online education. Unfortunately, many instructors and platforms do not adequately leverage learning analytics in their courses today. In this paper, we report on the value of these systems from the perspective of course instructors. We study these ideas through OARS, a modular and real-time learning analytics system that we deployed across more than ten online courses with tens of thousands of learners. We leverage this system as a starting point for semi-structured interviews with a diverse set of instructors. Our study suggests new design goals for learning analytics systems, the importance of real-time analytics to many instructors, and the value of flexibility in data selection and aggregation for an instructor when working with an analytics system.

References

[1]
Ryan S Baker. 2016. Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education 26, 2 (2016), 600--614.
[2]
Ryan S.J.d. Baker, Sidney K. D'Mello, Ma.Mercedes T. Rodrigo, and Arthur C. Graesser. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68, 4 (2010), 223 -- 241.
[3]
Sven Charleer, Jose Luis Santos, Joris Klerkx, and Erik Duval. 2014. Improving teacher awareness through activity, badge and content visualizations. In International Conference on Web-Based Learning. Springer, 143--152.
[4]
Doug Clow. 2012. The Learning Analytics Cycle: Closing the Loop Effectively. In Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge (LAK '12). 134--138.
[5]
Albert T. Corbett and John R. Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4 (01 Dec 1994), 253--278.
[6]
Anna L Dyckhoff. 2011. Implications for learning analytics tools: A meta-analysis of applied research questions. International Journal of Computer Information Systems and Industrial Management Applications 3, 1 (2011), 594--601.
[7]
Anna Lea Dyckhoff, Dennis Zielke, Mareike Bültmann, Mohamed Amine Chatti, and Ulrik Schroeder. 2012a. Design and implementation of a learning analytics toolkit for teachers. Journal of Educational Technology & Society 15, 3 (2012), 58.
[8]
Anna Lea Dyckhoff, Dennis Zielke, Mareike Bültmann, Mohamed Amine Chatti, and Ulrik Schroeder. 2012b. Design and implementation of a learning analytics toolkit for teachers. Journal of Educational Technology & Society 15, 3 (2012), 58.
[9]
Andrew J Elliot and Carol S Dweck. 2013. Handbook of competence and motivation. Guilford Publications.
[10]
Dragan Gasevic, Shane Dawson, Tim Rogers, and Danijela Gasevic. 2016. Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education 28 (2016), 68 -- 84.
[11]
G. Lopez C. Coleman J. Whitehill, J. Williams and J. Reich. 2015. Beyond prediction: First steps toward automatic intervention in MOOC student stopout. (2015).
[12]
Daniel A Keim. 2002. Information visualization and visual data mining. IEEE transactions on Visualization and Computer Graphics 8, 1 (2002), 1--8.
[13]
Kenneth R Koedinger, John R Anderson, William H Hadley, and Mary A Mark. 1997. Intelligent tutoring goes to school in the big city. (1997).
[14]
Kenneth R Koedinger, Ryan SJd Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper. 2010a. A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining 43 (2010).
[15]
Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2010b. The knowledge-learning-instruction (KLI) framework: Toward bridging the science-practice chasm to enhance robust student learning. Cognitive Science (2010).
[16]
Steven Lonn, Stephen J Aguilar, and Stephanie D Teasley. 2015. Investigating student motivation in the context of a learning analytics intervention during a summer bridge program. Computers in Human Behavior 47 (2015), 90--97.
[17]
Marsha Lovett, Oded Meyer, and Candace Thille. 2008. JIME-The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 2008, 1 (2008).
[18]
Learnosity Ltd. 2006--2018. Learnosity. Website. (2006--2018). Retrieved January 21, 2018 from www.learnosity.com.
[19]
Zachary A Pardos and Kevin Kao. 2015. moocRP: An open-source analytics platform. In Proceedings of the Second (2015) ACM conference on learning@ scale. ACM, 103--110.
[20]
Philip I Pavlik Jr, Hao Cen, and Kenneth R Koedinger. 2009. Performance Factors Analysis-A New Alternative to Knowledge Tracing. Online Submission (2009).
[21]
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep Knowledge Tracing. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (Eds.). Curran Associates, Inc., 505--513.
[22]
J. vanMarrienboer K. Yates R. Clark, D. Feldon and S. Early. 2008. Handbook of research on educational communications and technology (3rd ed.). Chapter Cognitive task analysis for training, 577--593.
[23]
Steve Ritter, Michael Yudelson, Stephen E Fancsali, and Susan R Berman. 2016. How mastery learning works at scale. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 71--79.
[24]
George Siemens, Dragan Gasevic, Caroline Haythornthwaite, Shane Peter Dawson, Simonbuckingham Shum, Rebecca Ferguson, Erik Duval, Katrien Verbert, Robert Baker, and others. 2011. Open Learning Analytics: an integrated & modularized platform. (2011).
[25]
S. Levine E. Brunskill T. Mandel, Y. Liu and Z. Popovic. 2014. Offline policy evaluation across representations with applications to educational games. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 1077--1084.
[26]
Kikumi K Tatsuoka. 1995. Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. Cognitively diagnostic assessment (1995), 327--359.
[27]
Dirk T Tempelaar, Bart Rienties, and Bas Giesbers. 2015. In search for the most informative data for feedback generation: Learning Analytics in a data-rich context. Computers in Human Behavior 47 (2015), 157--167.
[28]
J. Sewall V. Aleven, B. McLaren and K. Koedinger. 2006. The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. In Intelligent Tutoring Systems (ITS '06). 61--70.
[29]
Andrii Vozniuk, Sten Govaerts, and Denis Gillet. 2013. Towards portable learning analytics dashboards. In Advanced Learning Technologies (ICALT), 2013 IEEE 13th International Conference on. IEEE, 412--416.
[30]
Lisa Wang, Angela Sy, Larry Liu, and Chris Piech. 2017. Learning to represent student knowledge on programming exercises using deep learning. In Proceedings of the 10th International Conference on Educational Data Mining; Wuhan, China. 324--329.
[31]
Bronwyn Woods, David Adamson, Shayne Miel, and Elijah Mayfield. 2017. Formative Essay Feedback Using Predictive Scoring Models. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '17). ACM, New York, NY, USA, 2071--2080.

Cited By

View all
  • (2023)A checklist to guide the planning, designing, implementation, and evaluation of learning analytics dashboardsInternational Journal of Educational Technology in Higher Education10.1186/s41239-023-00394-620:1Online publication date: 3-May-2023
  • (2023)Designing for Student Understanding of Learning Analytics AlgorithmsArtificial Intelligence in Education10.1007/978-3-031-36272-9_43(528-540)Online publication date: 26-Jun-2023
  • (2020)Assessing Post-hoc Explainability of the BKT AlgorithmProceedings of the AAAI/ACM Conference on AI, Ethics, and Society10.1145/3375627.3375856(407-413)Online publication date: 7-Feb-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
L@S '18: Proceedings of the Fifth Annual ACM Conference on Learning at Scale
June 2018
391 pages
ISBN:9781450358866
DOI:10.1145/3231644
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. instructor-centered design
  2. learning analytics
  3. real-time systems

Qualifiers

  • Research-article

Conference

L@S '18
L@S '18: Fifth (2018) ACM Conference on Learning @ Scale
June 26 - 28, 2018
London, United Kingdom

Acceptance Rates

L@S '18 Paper Acceptance Rate 24 of 58 submissions, 41%;
Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)16
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)A checklist to guide the planning, designing, implementation, and evaluation of learning analytics dashboardsInternational Journal of Educational Technology in Higher Education10.1186/s41239-023-00394-620:1Online publication date: 3-May-2023
  • (2023)Designing for Student Understanding of Learning Analytics AlgorithmsArtificial Intelligence in Education10.1007/978-3-031-36272-9_43(528-540)Online publication date: 26-Jun-2023
  • (2020)Assessing Post-hoc Explainability of the BKT AlgorithmProceedings of the AAAI/ACM Conference on AI, Ethics, and Society10.1145/3375627.3375856(407-413)Online publication date: 7-Feb-2020
  • (2020)Reinforcement Learning for the Adaptive Scheduling of Educational ActivitiesProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376518(1-12)Online publication date: 21-Apr-2020
  • (2020)Including Learning Analytics in the Loop of Self-Paced Online Course Learning DesignInternational Journal of Artificial Intelligence in Education10.1007/s40593-020-00225-zOnline publication date: 9-Dec-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media