skip to main content
10.1145/3450613.3456818acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article

Characterising Student Engagement Modes through Low-Level Activity Patterns

Published: 21 June 2021 Publication History

Abstract

Existing approaches to characterise engagement in online learning focus on features of the interaction of students with the learning platform including the number of posts in forums, downloads of learning materials and time spent watching videos. However, little is known about what students actually do within the learning resources and whether these activities are indicators of learning outcomes. To bridge this gap, we associate low-level activity patterns with particular student engagement modes on a connectivist MOOC (cMOOC) that ran for four weeks and involved 224 students. Our findings indicate that our approach isolates meaningful interactive behavioural markers that are indicators of engagement, and are amenable to computation.

Supplementary Material

MP4 File (UMAP21-lp47.mp4)
Presentation video of "Characterising Student Engagement Modes through Low-Level Activity Patterns" paper. While the paper explores three research questions, the paper covers only one of them. Further details can be found in the paper.

References

[1]
M. Elena Alonso-Mencía, Carlos Alario-Hoyos, Jorge Maldonado-Mahauad, Iria Estévez-Ayres, Mar Pérez-Sanagustín, and Carlos Delgado Kloos. 2020. Self-regulated learning in MOOCs: lessons learned from a literature review. Educational Review 72, 3 (2020), 319–345. https://doi.org/10.1080/00131911.2019.1566208
[2]
Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2014. Engaging with Massive Online Courses. In Proceedings of the 23rd International Conference on World Wide Web (Seoul, Korea) (WWW ’14). Association for Computing Machinery, New York, NY, USA, 687–698. https://doi.org/10.1145/2566486.2568042
[3]
Ioannis Arapakis, Mounia Lalmas, and George Valkanas. 2014. Understanding Within-Content Engagement through Pattern Analysis of Mouse Gestures. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (Shanghai, China) (CIKM ’14). Association for Computing Machinery, New York, NY, USA, 1439–1448. https://doi.org/10.1145/2661829.2661909
[4]
Olatz Arbelaitz, Ibai Gurrutxaga, Javier Muguerza, Jesús M. Pérez, and Iñigo Perona. 2013. An extensive comparative study of cluster validity indices. Pattern Recognition 46, 1 (2013), 243 – 256. https://doi.org/10.1016/j.patcog.2012.07.021
[5]
Richard Atterer, Monika Wnuk, and Albrecht Schmidt. 2006. Knowing the User’s Every Move: User Activity Tracking for Website Usability Evaluation and Implicit Interaction. In WWW ’06. Edinburgh, Scotland, 203–212.
[6]
Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. 2011. Towards a science of user engagement (position paper). In WSDM workshop on user modelling for Web applications. 9–12.
[7]
John B Biggs. 2011. Teaching for quality learning at university: What the student does. McGraw-Hill Education.
[8]
Christopher Brooks, Craig Thompson, and Stephanie Teasley. 2015. A Time Series Interaction Analysis Method for Building Predictive Models of Learners Using Log Data. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (Poughkeepsie, New York) (LAK ’15). Association for Computing Machinery, New York, NY, USA, 126–135. https://doi.org/10.1145/2723576.2723581
[9]
Emma Brunskill, Dawn Zimmaro, and Candace Thille. 2018. Exploring the Impact of the Default Option on Student Engagement and Performance in a Statistics MOOC. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S ’18). Association for Computing Machinery, New York, NY, USA, Article 34, 4 pages. https://doi.org/10.1145/3231644.3231692
[10]
Mon Chu Chen, John R. Anderson, and Myeong Ho Sohn. 2001. What Can a Mouse Cursor Tell Us More? Correlation of Eye/Mouse Movements on Web Browsing. In CHI ’01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington) (CHI EA ’01). Association for Computing Machinery, New York, NY, USA, 281–282. https://doi.org/10.1145/634067.634234
[11]
Michelene T. H. Chi and Ruth Wylie. 2014. The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes. Educational Psychologist 49, 4 (2014), 219–243. https://doi.org/10.1080/00461520.2014.965823
[12]
Analía Cicchinelli, Eduardo Veas, Abelardo Pardo, Viktoria Pammer-Schindler, Angela Fessl, Carla Barreiros, and Stefanie Lindstädt. 2018. Finding Traces of Self-Regulated Learning in Activity Streams. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (Sydney, New South Wales, Australia) (LAK ’18). Association for Computing Machinery, New York, NY, USA, 191–200. https://doi.org/10.1145/3170358.3170381
[13]
Carleton Coffrin, Linda Corrin, Paula de Barba, and Gregor Kennedy. 2014. Visualizing Patterns of Student Engagement and Performance in MOOCs. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (Indianapolis, Indiana, USA) (LAK ’14). Association for Computing Machinery, New York, NY, USA, 83–92. https://doi.org/10.1145/2567574.2567586
[14]
Cody A. Coleman, Daniel T. Seaton, and Isaac Chuang. 2015. Probabilistic Use Cases: Discovering Behavioral Patterns for Predicting Certification. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15). Association for Computing Machinery, New York, NY, USA, 141–148. https://doi.org/10.1145/2724660.2724662
[15]
R. Wes Crues, Nigel Bosch, Michelle Perry, Lawrence Angrave, Najmuddin Shaik, and Suma Bhat. 2018. Refocusing the Lens on Engagement in MOOCs. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S ’18). Association for Computing Machinery, New York, NY, USA, Article 11, 10 pages. https://doi.org/10.1145/3231644.3231658
[16]
Samuel Dodson, Ido Roll, Matthew Fong, Dongwook Yoon, Negar M. Harandi, and Sidney Fels. 2018. An Active Viewing Framework for Video-Based Learning. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S ’18). Association for Computing Machinery, New York, NY, USA, Article 24, 4 pages. https://doi.org/10.1145/3231644.3231682
[17]
Claudia Ehmke and Stephanie Wilson. 2007. Identifying Web Usability Problems from Eye-Tracking Data. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI...but Not as We Know It - Volume 1 (University of Lancaster, United Kingdom) (BCS-HCI ’07). BCS Learning & Development Ltd., Swindon, GBR, 119–128.
[18]
Rebecca Ferguson and Doug Clow. 2015. Examining Engagement: Analysing Learner Subpopulations in Massive Open Online Courses (MOOCs). In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (Poughkeepsie, New York) (LAK ’15). Association for Computing Machinery, New York, NY, USA, 51–58. https://doi.org/10.1145/2723576.2723606
[19]
Ed Fincham, Alexander Whitelock-Wainwright, Vitomir Kovanović, Srećko Joksimović, Jan-Paul van Staalduinen, and Dragan Gašević. 2019. Counting Clicks is Not Enough: Validating a Theorized Model of Engagement in Learning Analytics. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (Tempe, AZ, USA) (LAK19). Association for Computing Machinery, New York, NY, USA, 501–510. https://doi.org/10.1145/3303772.3303775
[20]
Philip J. Guo, Juho Kim, and Rob Rubin. 2014. How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. In Proceedings of the First ACM Conference on Learning @ Scale Conference (Atlanta, Georgia, USA) (L@S ’14). Association for Computing Machinery, New York, NY, USA, 41–50. https://doi.org/10.1145/2556325.2566239
[21]
Qi Guo and Eugene Agichtein. 2010. Towards Predicting Web Searcher Gaze Position from Mouse Movements. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery, New York, NY, USA, 3601–3606. https://doi.org/10.1145/1753846.1754025
[22]
David M. Hilbert and David F. Redmiles. 2000. Extracting usability information from user interface events. Comput. Surveys 32, 4 (2000), 384–421.
[23]
Stacey L. Houston, Katherine Brady, Gayathri Narasimham, and Douglas Fisher. 2017. Pass the Idea Please: The Relationship between Network Position, Direct Engagement, and Course Performance in MOOCs. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge, Massachusetts, USA) (L@S ’17). Association for Computing Machinery, New York, NY, USA, 295–298. https://doi.org/10.1145/3051457.3054008
[24]
Jeff Huang, Ryen White, and Georg Buscher. 2012. User See, User Point: Gaze and Cursor Alignment in Web Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 1341–1350. https://doi.org/10.1145/2207676.2208591
[25]
Jeff Huang, Ryen W. White, and Susan Dumais. 2011. No Clicks, No Problem: Using Cursor Movements to Understand and Improve Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 1225–1234. https://doi.org/10.1145/1978942.1979125
[26]
Rosie Jones and Kristina Lisa Klinkner. 2008. Beyond the Session Timeout: Automatic Hierarchical Segmentation of Search Topics in Query Logs. In CIKM ’08. 699–708.
[27]
Youngho Kim, Ahmed Hassan, Ryen W. White, and Imed Zitouni. 2014. Modeling Dwell Time to Predict Click-Level Satisfaction. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (New York, New York, USA) (WSDM ’14). Association for Computing Machinery, New York, NY, USA, 193–202. https://doi.org/10.1145/2556195.2556220
[28]
René F. Kizilcec, Chris Piech, and Emily Schneider. 2013. Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (Leuven, Belgium) (LAK ’13). Association for Computing Machinery, New York, NY, USA, 170–179. https://doi.org/10.1145/2460296.2460330
[29]
Dmitry Lagun and Mounia Lalmas. 2016. Understanding User Attention and Engagement in Online News Reading. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (San Francisco, California, USA) (WSDM ’16). Association for Computing Machinery, New York, NY, USA, 113–122. https://doi.org/10.1145/2835776.2835833
[30]
Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of User Engagement. In User Modeling, Adaptation, and Personalization, Judith Masthoff, Bamshad Mobasher, Michel C. Desmarais, and Roger Nkambou (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 164–175.
[31]
Qiujie Li and Rachel Baker. 2016. Understanding Engagement in MOOCs. In Proceedings of the Ninth International Conference on Educational Data Mining. International Educational Data Mining Society, 605–606.
[32]
Jorge Maldonado-Mahauad, Mar Pérez-Sanagustín, René F. Kizilcec, Nicolás Morales, and Jorge Munoz-Gama. 2018. Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in Massive Open Online Courses. Computers in Human Behavior 80 (2018), 179 – 196. https://doi.org/10.1016/j.chb.2017.11.011
[33]
Colin Milligan, Allison Littlejohn, and Anoush Margaryan. 2013. Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching 9, 2 (June 2013), 149–159. http://oro.open.ac.uk/42259/ Special issue on ”Massive Open Online Courses”.
[34]
Benjamin Motz, Joshua Quick, Noah Schroeder, Jordon Zook, and Matthew Gunkel. 2019. The Validity and Utility of Activity Logs as a Measure of Student Engagement. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (Tempe, AZ, USA) (LAK19). Association for Computing Machinery, New York, NY, USA, 300–309. https://doi.org/10.1145/3303772.3303789
[35]
Eni Mustafaraj and Jessica Bu. 2015. The Visible and Invisible in a MOOC Discussion Forum. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15). Association for Computing Machinery, New York, NY, USA, 351–354. https://doi.org/10.1145/2724660.2728691
[36]
Vidhya Navalpakkam, LaDawn Jentzsch, Rory Sayres, Sujith Ravi, Amr Ahmed, and Alex Smola. 2013. Measurement and Modeling of Eye-Mouse Behavior in the Presence of Nonlinear Page Layouts. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13). Association for Computing Machinery, New York, NY, USA, 953–964. https://doi.org/10.1145/2488388.2488471
[37]
Heather L. O’Brien and Elaine G. Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology 59, 6 (2008), 938–955. https://doi.org/10.1002/asi.20801 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.20801
[38]
Heather L. O’Brien and Elaine G. Toms. 2010. The development and evaluation of a survey to measure user engagement. Journal of the American Society for Information Science and Technology 61, 1 (2010), 50–69. https://doi.org/10.1002/asi.21229 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.21229
[39]
Daan Odijk, Ryen W. White, Ahmed Hassan Awadallah, and Susan T. Dumais. 2015. Struggling and Success in Web Search. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (Melbourne, Australia) (CIKM ’15). Association for Computing Machinery, New York, NY, USA, 1551–1560. https://doi.org/10.1145/2806416.2806488
[40]
Daniel FO Onah, Jane Sinclair, and Russell Boyatt. 2014. Dropout rates of massive open online courses: behavioural patterns. Edulearn’14 (2014), 5825–5834.
[41]
Jihyun Park, Kameryn Denaro, Fernando Rodriguez, Padhraic Smyth, and Mark Warschauer. 2017. Detecting Changes in Student Behavior from Clickstream Data. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (Vancouver, British Columbia, Canada) (LAK ’17). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/3027385.3027430
[42]
Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume, and Lise Getoor. 2014. Uncovering Hidden Engagement Patterns for Predicting Learner Performance in MOOCs. In Proceedings of the First ACM Conference on Learning @ Scale Conference (Atlanta, Georgia, USA) (L@S ’14). Association for Computing Machinery, New York, NY, USA, 157–158. https://doi.org/10.1145/2556325.2567857
[43]
Amy L. Reschly and Sandra L. Christenson. 2012. Jingle, Jangle, and Conceptual Haziness: Evolution and Future Directions of the Engagement Construct. Springer US, Boston, MA, 3–19. https://doi.org/10.1007/978-1-4614-2018-7_1
[44]
Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information Processing & Management 24, 5 (1988), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0
[45]
Paul Thomas, Heather O’Brien, and Tom Rowlands. 2016. Measuring Engagement with Online Forms. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (Carrboro, North Carolina, USA) (CHIIR ’16). Association for Computing Machinery, New York, NY, USA, 325–328. https://doi.org/10.1145/2854946.2854988
[46]
Terry ; Chen Li Wang, Zhijun ; Anderson. 2018. How Learners Participate in Connectivist Learning: An Analysis of the Interaction Traces From a cMOOC. International Review of Research in Open and Distributed Learning 19, 1(2018). https://doi.org/10.19173/irrodl.v19i1.3269
[47]
Xu Wang, Miaomiao Wen, and Carolyn P. Rosé. 2016. Towards Triggering Higher-Order Thinking Behaviors in MOOCs. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (Edinburgh, United Kingdom) (LAK ’16). Association for Computing Machinery, New York, NY, USA, 398–407. https://doi.org/10.1145/2883851.2883964
[48]
Xu Wang, Diyi Yang, Miaomiao Wen, Kenneth Koedinger, and Carolyn P Rosé. 2015. Investigating How Student’s Cognitive Behavior in MOOC Discussion Forums Affect Learning Gains.International Educational Data Mining Society (2015).
[49]
Ran Yu, Ujwal Gadiraju, Peter Holtz, Markus Rokicki, Philipp Kemkes, and Stefan Dietze. 2018. Predicting User Knowledge Gain in Informational Search Sessions. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 75–84. https://doi.org/10.1145/3209978.3210064
[50]
Dongsong Zhang, Lina Zhou, Robert O Briggs, and Jay F Nunamaker Jr. 2006. Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information & management 43, 1 (2006), 15–27.
[51]
Mengxiao Zhu, Yoav Bergner, Yan Zhang, Ryan Baker, Yuan Wang, and Luc Paquette. 2016. Longitudinal Engagement, Performance, and Social Connectivity: A MOOC Case Study Using Exponential Random Graph Models. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (Edinburgh, United Kingdom) (LAK ’16). Association for Computing Machinery, New York, NY, USA, 223–230. https://doi.org/10.1145/2883851.2883934
[52]
Mengdie Zhuang, Gianluca Demartini, and Elaine G. Toms. 2017. Understanding Engagement through Search Behaviour. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (Singapore, Singapore) (CIKM ’17). Association for Computing Machinery, New York, NY, USA, 1957–1966. https://doi.org/10.1145/3132847.3132978

Cited By

View all
  • (2023)Multiclass Student Engagement Level Prediction using Belief-Rule Based Labelling2023 Sixth International Conference of Women in Data Science at Prince Sultan University (WiDS PSU)10.1109/WiDS-PSU57071.2023.00044(174-179)Online publication date: Mar-2023
  • (2023)Modeling Search Behavior Evolution on a Specialist Search EngineIT Professional10.1109/MITP.2023.324341325:2(30-35)Online publication date: 1-Mar-2023
  • (2022)Automatic modeling of student characteristics with interaction and physiological data using machine learning: A reviewFrontiers in Artificial Intelligence10.3389/frai.2022.10156605Online publication date: 3-Nov-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UMAP '21: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization
June 2021
325 pages
ISBN:9781450383660
DOI:10.1145/3450613
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Self-regulated online learning
  2. cMOOC
  3. engagement
  4. log mining

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • H2020 Research and Innovation Action

Conference

UMAP '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 162 of 633 submissions, 26%

Upcoming Conference

UMAP '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)42
  • Downloads (Last 6 weeks)3
Reflects downloads up to 07 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Multiclass Student Engagement Level Prediction using Belief-Rule Based Labelling2023 Sixth International Conference of Women in Data Science at Prince Sultan University (WiDS PSU)10.1109/WiDS-PSU57071.2023.00044(174-179)Online publication date: Mar-2023
  • (2023)Modeling Search Behavior Evolution on a Specialist Search EngineIT Professional10.1109/MITP.2023.324341325:2(30-35)Online publication date: 1-Mar-2023
  • (2022)Automatic modeling of student characteristics with interaction and physiological data using machine learning: A reviewFrontiers in Artificial Intelligence10.3389/frai.2022.10156605Online publication date: 3-Nov-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media