Abstract
The past few years have seen a surge of interest in deep neural networks. The wide application of deep learning in other domains such as image classification has driven considerable recent interest and efforts in applying these methods in educational domains. However, there is still limited research comparing the predictive power of the deep learning approach with the traditional feature engineering approach for common student modeling problems such as sensor-free affect detection. This paper aims to address this gap by presenting a thorough comparison of several deep neural network approaches with a traditional feature engineering approach in the context of affect and behavior modeling. We built detectors of student affective states and behaviors as middle school students learned science in an open-ended learning environment called Betty’s Brain, using both approaches. Overall, we observed a tradeoff where the feature engineering models were better when considering a single optimized threshold (for intervention), whereas the deep learning models were better when taking model confidence fully into account (for discovery with models analyses).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
60-s clip variants were initially tested but were less effective than 20-s clips.
- 2.
A 20-s clip was also tested for the deep learning models, but it did not work as well as the 60-s clips.
References
Clarke-Midura, J., Yudelson, M.V.: Towards identifying students’ causal reasoning using machine learning. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 704–707. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_93
Rowe, E., Asbell-Clarke, J., Baker, R.S., Eagle, M., Hicks, A.G., Barnes, T.M., Brown, R.A., Edwards, T.: Assessing implicit science learning in digital games. Comput. Hum. Behav. 76, 617–630 (2017)
Baker, R.S., Gowda, S.M., Wixon, M., Kalka, J., Wagner, A.Z., Salvi, A., Aleven, V., Kusbit, G.W., Ocumpaugh, J., Rossi, L.: Towards sensor-free affect detection in cognitive tutor algebra. In: Proceedings of the 5th International Conference on Educational Data Mining, pp. 126–133 (2012)
D’Mello, S., Jackson, T., Craig, S., Morgan, B., Chipman, P., White, H., Person, N., Kort, B., Kaliouby, R.e., Picard, R., Graesser, A.: AutoTutor detects and responds to learners affective and cognitive states. In: Proceedings of the Workshop on Emotional and Cognitive Issues in ITS in Conjunction with the 9th International Conference on ITS, pp. 31–43 (2008)
Pardos, Z.A., Baker, R.S., Pedro, M.O.C.Z.S., Gowda, S.M., Gowda, S.M.: Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes. In: Proceedings of the 3rd International Conference on Learning Analytics and Knowledge, pp. 117–124 (2013)
Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. In: Proceedings of the 2009 Conference on Artificial Intelligence in Education (AIED 2009), pp. 17–24. IOS Press, Amsterdam (2009)
Baker, R.S., Corbett, A.T., Roll, I., Koedinger, K.R.: Developing a generalizable detector of when students game the system. User Model. User-Adapt. Interact. 18, 287–314 (2008)
Cetintas, S., Si, L., Xin, Y.P., Hord, C.: Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Trans. Learn. Technol. 3, 228–236 (2010)
Kai, S., Paquette, L., Baker, R.S., Bosch, N., D’Mello, S., Ocumpaugh, J., Shute, V., Ventura, M.: A comparison of video-based and interaction-based affect detectors in physics playground. In: Proceedings of the 8th International Conference on Educational Data Mining, pp. 77–84 (2015)
Ocumpaugh, J., Baker, R.S., Rodrigo, M.M.T.: Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 technical and training manual. Technical report, Teachers College, Columbia University, Ateneo Laboratory for the Learning Sciences (2015)
Fancsali, S.E.: Causal discovery with models: behavior, affect, and learning in cognitive tutor algebra. In: Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), pp. 28–35 (2014)
San Pedro, M.O.Z., Baker, R.S., Bowers, A.J., Heffernan, N.T.: Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In: Proceedings of the 6th International Conference on Educational Data Mining, pp. 177–184 (2013)
San Pedro, M.O.Z., Snow, E.L., Baker, R.S., McNamara, D.S., Heffernan, N.T.: Exploring Dynamic Assessments of Affect, Behavior, and Cognition and Math State Test Achievement. In: Proceedings of the 8th International Conference on Educational Data Mining, pp. 85–92 (2015)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Botelho, A.F., Baker, R.S., Heffernan, N.T.: Improving sensor-free affect detection using deep learning. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 40–51. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_4
Khajah, M., Lindsey, R.V., Mozer, M.C.: How deep is knowledge tracing? In: Proceedings of the 9th International Conference on Educational Data Mining (EDM 2016), pp. 94–101 (2016)
Lin, C., Chi, M.: A comparisons of BKT, RNN and LSTM for learning gain prediction. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 536–539. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_58
Piech, C., Spencer, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., Sohl-Dickstein, J.: Deep knowledge tracing. In: Advances in Neural Information Processing Systems (NIPS 2015), vol. 28, pp. 505–513. Curran Associates, Inc. (2015)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, pp. 1097–1105, Lake Tahoe, Nevada (2012)
Botelho, A.F., Baker, R.S., Heffernan, N.T.: Developing and evaluating “deep” sensor-free detectors of student affect. (Manuscript in preparation)
Leelawong, K., Biswas, G.: Designing learning by teaching agents: the Betty’s brain system. Int J. Artif. Intell. Educ. 18, 181–208 (2008)
Baker, R.S., Ocumpaugh, J., Gowda, S.M., Kamarainen, A.M., Metcalf, S.J.: Extending log-based affect detection to a multi-user virtual environment for science. In: Proceedings of the 22nd Conference on User Modelling, Adaptation, and Personalization, pp. 290–300 (2014)
Ocumpaugh, J., Baker, R.S., Rodrigo, M.M., Salvi, A., Velsen, M.V., Aghababyan, A., Martin, T.: HART: the human affect recording tool. In: Proceedings of the 33rd Annual International Conference on the Design of Communication (SIGDOC 2015). ACM, New York (2015)
Mierswa, I., Scholz, M., Klinkenberg, R., Wurst, M., Euler, T.: Rapid prototyping for complex data mining tasks. In: Proceedings of KDD 2006, pp. 935–940 (2006)
Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by Exponential Linear Units (ELUs). In: ICLR 2016 (2016)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Paquette, L., Baker, R.S., Pedro, M.A.S., Gobert, J.D., Rossi, L., Nakama, A., Kauffman-Rogoff, Z.: Sensor-free affect detection for a simulation-based science inquiry learning environment. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems, pp. 1–10 (2014)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112. Curran Associates, Inc. (2014)
Ocumpaugh, J., Baker, R.S., Gowda, S.M., Heffernan, N.T., Heffernan, C.: Population validity for educational data mining: a case study in affect detection. Br. J. Educ. Psychol. 45, 487–501 (2014)
Acknowledgments
We would like to thank the National Science Foundation (NSF) for their support (#DRL-1561567).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Jiang, Y. et al. (2018). Expert Feature-Engineering vs. Deep Neural Networks: Which Is Better for Sensor-Free Affect Detection?. In: Penstein Rosé, C., et al. Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science(), vol 10947. Springer, Cham. https://doi.org/10.1007/978-3-319-93843-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-93843-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-93842-4
Online ISBN: 978-3-319-93843-1
eBook Packages: Computer ScienceComputer Science (R0)