skip to main content
10.1145/3375462.3375485acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
short-paper

#Confused and beyond: detecting confusion in course forums using students' hashtags

Published: 23 March 2020 Publication History

Abstract

Students' confusion is a barrier for learning, contributing to loss of motivation and to disengagement with course materials. However, detecting students' confusion in large-scale courses is both time and resource intensive. This paper provides a new approach for confusion detection in online forums that is based on harnessing the power of students' self-reported affective states (reported using a set of pre-defined hashtags). It presents a rule for labeling confusion, based on students' hashtags in their posts, that is shown to align with teachers' judgement. We use this labeling rule to inform the design of an automated classifier for confusion detection for the case when there are no self-reported hashtags present in the test set. We demonstrate this approach in a large scale Biology course using the Nota Bene annotation platform. This work lays the foundation to empower teachers with better support tools for detecting and alleviating confusion in online courses.

References

[1]
Akshay Agrawal and Andreas Paepcke. 2014. The Stanford MOOCPosts Dataset.
[2]
Akshay Agrawal, Jagadish Venkatraman, Shane Leonard, and Andreas Paepcke. 2015. YouEDU: addressing confusion in MOOC discussion forums by recommending instructional video clips. (2015).
[3]
Giora Alexandron, José A Ruipérez-Valiente, Zhongzhou Chen, Pedro J Muñoz-Merino, and David E Pritchard. 2017. Copying@Scale: Using harvesting accounts for collecting correct answers in a MOOC. Computers & Education 108 (2017), 96--114.
[4]
Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced sentiment learning using twitter hashtags and smileys. In Proceedings of the 23rd international conference on computational linguistics: posters. Association for Computational Linguistics, 241--249.
[5]
Sidney D' Mello and Art Graesser. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 2 (2012), 145--157.
[6]
Sidney D' Mello, Blair Lehman, Reinhard Pekrun, and Art Graesser. 2014. Confusion can be beneficial for learning. Learning and Instruction 29 (2014), 153--170.
[7]
Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. arXiv preprint arXiv:1609.08359 (2016).
[8]
Beate Grawemeyer, Manolis Mavrikis, Wayne Holmes, and Sergio Gutierrez-Santos. 2015. Adapting feedback types according to students' affective states. In International Conference on Artificial Intelligence in Education. Springer, 586--590.
[9]
Diane Marie C Lee, Ma Mercedes T Rodrigo, Ryan SJ d Baker, Jessica O Sugay, and Andrei Coronel. 2011. Exploring the relationship between novice programmer confusion and achievement. In International conference on affective computing and intelligent interaction. Springer, 175--184.
[10]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
[11]
Kelly Miller, Sacha Zyto, David Karger, Junehee Yoo, and Eric Mazur. 2016. Analysis of student engagement in an online annotation system in the context of a flipped introductory physics class. Physical Review Physics Education Research 12, 2 (2016), 020143.
[12]
Saif M Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence 31, 2 (2015), 301--326.
[13]
Reinhard Pekrun, Thomas Goetz, Wolfram Titz, and Raymond P Perry. 2002. Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational psychologist 37, 2 (2002), 91--105.
[14]
Ma Mercedes T Rodrigo, Ryan SJd Baker, and Julieta Q Nabos. 2010. The relationships between sequences of affective states and learner achievement. In Proceedings of the 18th International Conference on Computers in Education. 56--60.
[15]
John A Ross. 2006. The reliability, validity, and utility of self-assessment. (2006).
[16]
Peter D Turney. 2005. Measuring semantic similarity by latent relational analysis. arXiv preprint cs/0508053 (2005).
[17]
Xiaocong Wei, Hongfei Lin, Liang Yang, and Yuhai Yu. 2017. Aconvolution-LSTM-based deep neural network for cross-domain MOOC forum post classification. Information 8, 3 (2017), 92.
[18]
Beverly Woolf, Winslow Burleson, Ivon Arroyo, Toby Dragon, David Cooper, and Rosalind Picard. 2009. Affect-aware tutors: recognising and responding to student affect. International Journal of Learning Technology 4, 3-4 (2009), 129--164.
[19]
Diyi Yang, Miaomiao Wen, Iris Howley, Robert Kraut, and Carolyn Rose. 2015. Exploring the effect of confusion in discussion forums of massive open online courses. In Proceedings of the second (2015) ACM conference on learning@ scale. ACM, 121--130.
[20]
Ziheng Zeng, Snigdha Chaturvedi, and Suma Bhat. 2017. Learner Affect Through the Looking Glass: Characterization and Detection of Confusion in Online Courses. In EDM.
[21]
Amy X Zhang, Michele Igo, Marc Facciotti, and David Karger. 2017. Using student annotated hashtags and emojis to collect nuanced affective states. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. ACM, 319--322.
[22]
Sacha Zyto, David Karger, Mark Ackerman, and Sanjoy Mahajan. 2012. Successful classroom deployment of a social document annotation system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1883--1892.

Cited By

View all
  • (2024)#let’s-discuss: Analyzing Students’ Use of Emoji when Interacting with Course ReadingsInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00401-5Online publication date: 8-May-2024
  • (2023)MeetScript: Designing Transcript-based Interactions to Support Active Participation in Group Video MeetingsProceedings of the ACM on Human-Computer Interaction10.1145/36101967:CSCW2(1-32)Online publication date: 4-Oct-2023
  • (2023)Is the Latest the Greatest? A Comparative Study of Automatic Approaches for Classifying Educational Forum PostsIEEE Transactions on Learning Technologies10.1109/TLT.2022.322701316:3(339-352)Online publication date: 1-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
LAK '20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge
March 2020
679 pages
ISBN:9781450377126
DOI:10.1145/3375462
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 March 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. confusion detection
  2. emojis
  3. hashtags
  4. online discussion forum
  5. self-reported affect
  6. text classification

Qualifiers

  • Short-paper

Funding Sources

  • National Science Foundation IUSE
  • Israeli Ministry of Science, Technology and Space

Conference

LAK '20

Acceptance Rates

LAK '20 Paper Acceptance Rate 80 of 261 submissions, 31%;
Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)35
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)#let’s-discuss: Analyzing Students’ Use of Emoji when Interacting with Course ReadingsInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00401-5Online publication date: 8-May-2024
  • (2023)MeetScript: Designing Transcript-based Interactions to Support Active Participation in Group Video MeetingsProceedings of the ACM on Human-Computer Interaction10.1145/36101967:CSCW2(1-32)Online publication date: 4-Oct-2023
  • (2023)Is the Latest the Greatest? A Comparative Study of Automatic Approaches for Classifying Educational Forum PostsIEEE Transactions on Learning Technologies10.1109/TLT.2022.322701316:3(339-352)Online publication date: 1-Jun-2023
  • (2023)Leveraging explainability for discussion forum classification: Using confusion detection as an exampleDistance Education10.1080/01587919.2022.2150145(1-16)Online publication date: 2-Jan-2023
  • (2022)Leveraging Class Balancing Techniques to Alleviate Algorithmic Bias for Predictive Tasks in EducationIEEE Transactions on Learning Technologies10.1109/TLT.2022.319627815:4(481-492)Online publication date: 1-Aug-2022
  • (2021)Addressing Data Scarcity in Multimodal User State Recognition by Combining Semi-Supervised and Supervised LearningCompanion Publication of the 2021 International Conference on Multimodal Interaction10.1145/3461615.3486575(317-323)Online publication date: 18-Oct-2021
  • (2021)Detecting Disruptive Talk in Student Chat-Based Discussion within Collaborative Game-Based Learning EnvironmentsLAK21: 11th International Learning Analytics and Knowledge Conference10.1145/3448139.3448178(405-415)Online publication date: 12-Apr-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media