Abstract
Hand-raising signals students’ willingness to participate actively in the classroom discourse. It has been linked to academic achievement and cognitive engagement of students and constitutes an observable indicator of behavioral engagement. However, due to the large amount of effort involved in manual hand-raising annotation by human observers, research on this phenomenon, enabling teachers to understand and foster active classroom participation, is still scarce. An automated detection approach of hand-raising events in classroom videos can offer a time- and cost-effective substitute for manual coding. From a technical perspective, the main challenges for automated detection in the classroom setting are diverse camera angles and student occlusions. In this work, we propose utilizing and further extending a novel view-invariant, occlusion-robust machine learning approach with long short-term memory networks for hand-raising detection in classroom videos based on body pose estimation. We employed a dataset stemming from 36 real-world classroom videos, capturing 127 students from grades 5 to 12 and 2442 manually annotated authentic hand-raising events. Our temporal model trained on body pose embeddings achieved an \(F_{1}\) score of 0.76. When employing this approach for the automated annotation of hand-raising instances, a mean absolute error of 3.76 for the number of detected hand-raisings per student, per lesson was achieved. We demonstrate its application by investigating the relationship between hand-raising events and self-reported cognitive engagement, situational interest, and involvement using manually annotated and automatically detected hand-raising instances. Furthermore, we discuss the potential of our approach to enable future large-scale research on student participation, as well as privacy-preserving data collection in the classroom context.
B. Bühler and R. Hou—Both authors contributed equally.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ahuja, K., et al.: Edusense: practical classroom sensing at scale. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 3(3), 1–26 (2019)
Böheim, R., Knogler, M., Kosel, C., Seidel, T.: Exploring student hand-raising across two school subjects using mixed methods: an investigation of an everyday classroom behavior from a motivational perspective. Learn. Instr. 65, 101250 (2020)
Böheim, R., Urdan, T., Knogler, M., Seidel, T.: Student hand-raising as an indicator of behavioral engagement and its role in classroom learning. Contemp. Educ. Psychol. 62, 101894 (2020)
Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., Sheikh, Y.A.: OpenPose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43, 172–186 (2019)
Dutta, A., Zisserman, A.: The via annotation software for images, audio and video. In: ACM International Conference on Multimedia, pp. 2276–2279 (2019)
Frank, B.: Presence messen in laborbasierter Forschung mit Mikrowelten: Entwicklung und erste Validierung eines Fragebogens zur Messung von Presence. Springer, Heidelberg (2014)
Goldberg, P., et al.: Attentive or not? Toward a machine learning approach to assessing students’ visible engagement in classroom instruction. Educ. Psychol. Rev. 33, 27–49 (2021)
Zhou, H., Jiang, F., Shen, R.: Who are raising their hands? Hand-raiser seeking based on object detection and pose estimation. In: Asian Conference on Machine Learning, pp. 470–485 (2018)
Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2014)
Knogler, M., Harackiewicz, J.M., Gegenfurtner, A., Lewalter, D.: How situational is situational interest? Investigating the longitudinal structure of situational interest. Contemp. Educ. Psychol. 43, 39–50 (2015)
Liao, W., Xu, W., Kong, S., Ahmad, F., Liu, W.: A two-stage method for hand-raising gesture recognition in classroom. In: International Conference on Educational and Information Technology. ACM (2019)
Lin, F.C., Ngo, H.H., Dow, C.R., Lam, K.H., Le, H.L.: Student behavior recognition system for the classroom environment based on skeleton pose estimation and person detection. Sensors 21(16), 5314 (2021)
Liu, T., et al.: View-invariant, occlusion-robust probabilistic embedding for human pose. Int. J. Comput. Vis. 130(1), 111–135 (2022)
Nguyen, P.D., et al.: A new dataset and systematic evaluation of deep learning models for student activity recognition from classroom videos. In: International Conference on Multimedia Analysis and Pattern Recognition. IEEE (2022)
Rimm-Kaufman, S.E., Baroody, A.E., Larsen, R.A., Curby, T.W., Abry, T.: To what extent do teacher-student interaction quality and student gender contribute to fifth graders’ engagement in mathematics learning? J. Educ. Psychol. 107(1), 170 (2015)
Sedova, K., et al.: Do those who talk more learn more? The relationship between student classroom talk and student achievement. Learn. Instr. 63, 101217 (2019)
Si, J., Lin, J., Jiang, F., Shen, R.: Hand-raising gesture detection in real classrooms using improved R-FCN. Neurocomputing 359, 69–76 (2019)
Liu, T., Jiang, F., Shen, R.: Fast and accurate hand-raising gesture detection in classroom. In: Yang, H., Pasupa, K., Leung, A.C.-S., Kwok, J.T., Chan, J.H., King, I. (eds.) ICONIP 2020. CCIS, vol. 1332, pp. 232–239. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63820-7_26
Yu-Te, K., Han-Yen, Y., Yi-Chi, C.: A classroom atmosphere management system for analyzing human behaviors in class activities. In: International Conference on Artificial Intelligence in Information and Communication. IEEE (2019)
Zhang, S., Liu, X., Xiao, J.: On geometric features for skeleton-based action recognition using multilayer LSTM networks. In: IEEE Winter Conference on Applications of Computer Vision, pp. 148–157 (2017)
Jie, Y., Cooperstock, J.R.: Arm gesture detection in a classroom environment. In: Sixth IEEE Workshop on Applications of Computer Vision (2002). ISBN 0769518583
Bo, N.B., van Hese, P., van Cauwelaert, D., Veelaert, P., Philips, W.: Detection of a hand-raising gesture by locating the arm. In: IEEE International Conference on Robotics and Biomimetics (2011). ISBN 9781457721380
Acknowledgements
Babette Bühler is a doctoral candidate and supported by the LEAD Graduate School and Research Network, which is funded by the Ministry of Science, Research and the Arts of the state of Baden-Württemberg within the framework of the sustainability funding for the projects of the Excellence Initiative II. Efe Bozkir and Enkelejda Kasneci acknowledge the funding by the DFG with EXC number 2064/1 and project number 390727645. This work is also supported by Leibniz-WissenschaftsCampus Tübingen “Cognitive Interfaces” by a grant to Ulrich Trautwein, Peter Gerjets, and Enkelejda Kasneci. We thank Katrin Kunz and Jan Thiele for their excellent assistance.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bühler, B. et al. (2023). Automated Hand-Raising Detection in Classroom Videos: A View-Invariant and Occlusion-Robust Machine Learning Approach. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-36272-9_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36271-2
Online ISBN: 978-3-031-36272-9
eBook Packages: Computer ScienceComputer Science (R0)