Skip to main content

Detection of Subject Attention in an Active Environment Through Facial Expressions Using Deep Learning Techniques and Computer Vision

  • Conference paper
  • First Online:
Advances in Neuroergonomics and Cognitive Engineering (AHFE 2020)

Abstract

This research aims for investigation of workers in an industrial environment and can be used as an alternate for monitoring an attention of operator in real-time. Detection of attentiveness and non-attentiveness of people working in an industry could help to identify the weaknesses and strengths of any industrial organization. Human factor is the main and the most critical part of any industrial organization. As a special case, we have established how to detect student attention in the classroom using deep learning techniques along with computer vision. Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) are used to extract the percentage of attentiveness and non-attentiveness of students based on the student emotions in the classroom. We used the FER-2013 data set for this paper. As per the study, human has finite number of emotions. So, it is easy if we include some emotions in an attentive (Happy, Anger, Surprised and Neutral) domain and some emotions in non-attentive (Sad, Fear and Disgust) domain. This will help the teacher in a way that he can easily evaluate his class attentiveness. On another side, it is also the evaluation of the teacher’s teaching methodology because if the students are engaged in his lecture it means his teaching methodology is good and if most of the students are not engaged then the teacher needs to revise his methodology of teaching in order to engage his class during the lecture.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Park, S.: Student engagement and classroom variables in improving mathematics achievement. Asia Pacific Educ. Rev. 6, 87–97 (2005). https://doi.org/10.1007/BF03024970

    Article  Google Scholar 

  2. Kraushaar, J.M., Novak, D.C.: Examining the affects of student multitasking with laptops during the lecture. J. Inf. Syst. Educ. 21(2) (2019). Article 11

    Google Scholar 

  3. Nicole: Kaggle, 02 April 2019. https://www.kaggle.com/nicolejyt/facialexpressionrecognition

  4. Krithika, L.B., Lakshmi Priya, G.G.: Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Comput. Sci. 85(Cms), 767–776 (2016)

    Article  Google Scholar 

  5. Martin, J., Torres, A.: User’s Guide and Toolkit for the Surveys of Student Engagement: The High School Survey of Student Engagement (HSSSE) and the Middle Grades Survey of Student Engagement (MGSSE) I. What is Student Engagement and Why is it Important? (2016). http://www.isbe.net/learningsupports/pdfs/engagement-concept.pdf. Accessed 04 Oct 2019

  6. Skinner, E.A., Pitzer, J.R.: Developmental dynamics of student engagement, coping, and everyday resilience. In: Christenson, S., Reschly, A., Wylie, C. (eds.) Handbook of Research on Student Engagement, pp. 21–44. Springer, Boston (2012). https://doi.org/10.1007/978-1-4614-2018-7_2

    Chapter  Google Scholar 

  7. Hobsons.com: How to Increase Student Engagement at your School A Guide to Connecting Learning and Life Through Three Dimensions of Student Engagement 2 About Hobsons (2013)

    Google Scholar 

  8. Zaletelj, J., Košir, A.: Predicting students’ attention in the classroom from Kinect facial and body features. Eurasip. J. Image Video Process. 2017(1), 80 (2017)

    Article  Google Scholar 

  9. Veliyath, N., De, P., Allen, A.A., Hodges, C.B., Mitra, A.: Modeling students’ attention in the classroom using eyetrackers. In: ACMSE 2019 – Proceedings of the 2019 ACM Southeast Conference, pp. 2–9 (2019)

    Google Scholar 

  10. Salman, A., et al.: Real-time fish detection in complex backgrounds using probabilistic background modelling. Ecol. Inform. 51, 44–51 (2019)

    Article  Google Scholar 

  11. Afzal, M.Z., et al.: Document image binarization using LSTM: a sequence learning approach. In: Proceedings of the 3rd International Workshop on Historical Document Imaging and Processing (2015)

    Google Scholar 

  12. Siddiqui, S.A., et al.: Automatic fish species classification in underwater videos: exploiting pre-trained deep neural network models to compensate for limited labelled data. ICES J. Mar. Sci. 75(1), 374–389 (2018)

    Article  Google Scholar 

  13. Uzair, M., et al.: Representation learning with deep extreme learning machines for efficient image set classification. Neural Comput. Appl. 30(4), 1211–1223 (2018)

    Article  Google Scholar 

  14. Van den Oord, A., Dieleman, S., Schrauwen, B.: Deep content-based music recommendation. In: Advances in Neural Information Processing Systems (2013)

    Google Scholar 

  15. Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning (2008)

    Google Scholar 

  16. Huang, Y., Wang, W., Wang, L.: Instance-aware image and sentence matching with selective multimodal LSTM. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  17. Sak, H., Senior, A., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, pp. 338–342 (2014)

    Google Scholar 

  18. Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 855–868 (2009). https://doi.org/10.1109/TPAMI.2008.137

    Article  Google Scholar 

Download references

Acknowledgement

We would like to acknowledge School of Electrical Engineering and Computer Sciences (SEECS), National University of Sciences and Technology (NUST), Islamabad, for technical support and funding.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Naqash Gerard .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gerard, N. et al. (2021). Detection of Subject Attention in an Active Environment Through Facial Expressions Using Deep Learning Techniques and Computer Vision. In: Ayaz, H., Asgher, U. (eds) Advances in Neuroergonomics and Cognitive Engineering. AHFE 2020. Advances in Intelligent Systems and Computing, vol 1201. Springer, Cham. https://doi.org/10.1007/978-3-030-51041-1_43

Download citation

Publish with us

Policies and ethics