Skip to main content
SpringerLink
Log in

An Application of Machine Learning and Image Processing to Automatically Detect Teachers’ Gestures

  • Conference paper
  • First Online:
Advances in Computational Collective Intelligence (ICCCI 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1287))

Included in the following conference series:

Abstract

Providing teachers with detailed feedback about their gesticulation in class requires either one-on-one expert coaching, or highly trained observers to hand code classroom recordings. These methods are time consuming, expensive and require considerable human expertise, making them very difficult to scale to large numbers of teachers. Applying Machine Learning and Image processing we develop a non-invasive detector of teachers’ gestures. We use a multi-stage approach for the spotting task. Lessons recorded with a standard camera are processed offline with the OpenPose software. Next, using a gesture classifier trained on a previous training set with Machine Learning, we found that on new lessons the precision rate is between 54 and 78%. The accuracy depends on the training and testing datasets that are used. Thus, we found that using an accessible, non-invasive and inexpensive automatic gesture recognition methodology, an automatic lesson observation tool can be implemented that will detect possible teachers’ gestures. Combined with other technologies, like speech recognition and text mining of the teacher discourse, a powerful and practical tool can be offered to provide private and timely feedback to teachers about communication features of their teaching practices.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sheu, F.R., Chen, N.S.: Taking a signal: A review of gesture-based computing research in education. Comput. Educ. (2014). https://doi.org/10.1016/j.compedu.2014.06.008

    Article  Google Scholar 

  2. Patrona, F., Chatzitofis, A., Zarpalas, D., Daras, P.: Motion analysis: action detection, recognition and evaluation based on motion capture data. Pattern Recognit. 76, 612–622 (2018). https://doi.org/10.1016/j.patcog.2017.12.007

    Article  Google Scholar 

  3. Johnson, L., Levine, A., Smith, R., Stone, S.: The Horizon Report 2010. ERIC (2010)

    Google Scholar 

  4. Johnson, L., Adams, S., Cummins, M.: Horizon Report 2012 K-12 Edition (2012)

    Google Scholar 

  5. Farsani, D.: Making Multi-Modal Mathematical Meaning in Multilingual Classrooms. Thesis (2015)

    Google Scholar 

  6. Araya, R., Farsani, D., Hernández, J.: How to attract students’ visual attention. In: Verbert, K., Sharples, M., Klobučar, T. (eds.) EC-TEL 2016. LNCS, vol. 9891, pp. 30–41. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45153-4_3

    Chapter  Google Scholar 

  7. Farsani, D.: Complementary functions of learning mathematics in complementary schools. In: Halai, A., Clarkson, P. (eds.) Teaching and Learning Mathematics in Multilingual Classrooms. LNCS, pp. 227–247. SensePublishers, Rotterdam (2016). https://doi.org/10.1007/978-94-6300-229-5_15

    Chapter  Google Scholar 

  8. Ramseyer, F., Tschacher, W.: Nonverbal synchrony in psychotherapy: coordinated body movement reflects relationship quality and outcome. J. Consult. Clin. Psychol. (2011). https://doi.org/10.1037/a0023419

    Article  Google Scholar 

  9. Farsani, D., Barraza, P., Martinic, S.: Non-verbal synchrony as a pedagogical tool for medical education: a clinical case study. Kuwait Med. J. (accepted)

    Google Scholar 

  10. Farsani, D.: Mathematics education and the analysis of language working group: making multimodal mathematical meaning. Br. Soc. Res. Learn. Math. 32, 19–24 (2012)

    Google Scholar 

  11. McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago, IL, US (1992)

    Google Scholar 

  12. Kendon, A.: Some relationships between body motion and speech. Stud. Dyadic Commun. 7(177), 90 (1972). https://doi.org/10.1016/B978-0-08-015867-9.50013-7

    Article  Google Scholar 

  13. Sfard, A.: What’s all the fuss about gestures? A commentary. Educ. Stud. Math. 70, 191–200 (2009). https://doi.org/10.1007/s10649-008-9161-1

    Article  Google Scholar 

  14. Maschietto, M., Bussi, M.G.B.: Meaning construction through semiotic means: the case of the visual pyramid. Int. Gr. Psychol. Math. Educ. 3, 313–320 (2005)

    Google Scholar 

  15. Streeck, J.: The significance of gesture: how it is established. IPrA Pap. Pragmat. (1988). https://doi.org/10.1075/iprapip.2.1-2.03str

    Article  Google Scholar 

  16. Kendon, A.: Gesticulation and speech: two aspects of the process of uterance in M. Group. The relationship of the verbal and nonverbal communication. The Hague: Mouton, pp. 207–228 (1980)

    Google Scholar 

  17. Kendon, A.: Do gestures communicate?: a review. Res. Lang. Soc. Interact. (1994). https://doi.org/10.1207/s15327973rlsi2703_2

    Article  Google Scholar 

  18. Thompson, L.A., Massaro, D.W.: Children′s integration of speech and pointing gestures in comprehension. J. Exp. Child Psychol. (1994). https://doi.org/10.1006/jecp.1994.1016

    Article  Google Scholar 

  19. Graham, J.A., Heywood, S.: The effects of elimination of hand gestures and of verbal codability on speech performance. Eur. J. Soc. Psychol. (1975). https://doi.org/10.1002/ejsp.2420050204

    Article  Google Scholar 

  20. Mcneil, N.M., Alibali, M.W., Evans, J.L.: The role of gesture in children’s comprehension of spoken language: now they need it, now they don’t. J. Nonverbal Behav. 24, 131–150 (2000). https://doi.org/10.1023/A:1006657929803

    Article  Google Scholar 

  21. Riseborough, M.G.: Physiographic gestures as decoding facilitators: three experiments exploring a neglected facet of communication. J. Nonverbal Behav. (1981). https://doi.org/10.1007/BF00986134

    Article  Google Scholar 

  22. Alibali, M.W., Nathan, M.J.: Teachers’ gestures as a means of scaffolding students’ understanding: evidence from an early algebra lesson. Video Res. Learn. Sci. 349–365 (2007)

    Google Scholar 

  23. Pea, R., Lindgren, R.: Collaboration design patterns in uses of a video platform for research and education. IEEE Trans. Learn. Technol. 1, 235–247 (2008). https://doi.org/10.1109/TLT.2009.5

    Article  Google Scholar 

  24. Richland, L.E., Zur, O., Holyoak, K.J.: Cognitive supports for analogies in the mathematics classroom. Sci. 316, 1128–1129 (2007). https://doi.org/10.1126/science.1142103

    Article  Google Scholar 

  25. Hiebert, J., et al.: Teaching mathematics in seven countries: Results from the TIMSS 1999 video study. Educ. Stat. Q. 5, 7–15 (2003)

    Google Scholar 

  26. Richland, L.E., Hansen, J.: Reducing cognitive load in learning by analogy. Int. J. Psychol. Stud. 5, 1–11 (2013). https://doi.org/10.5539/ijps.v5n4p69

    Article  Google Scholar 

  27. Valenzeno, L., Alibali, M.W., Klatzky, R.: Teachers’ gestures facilitate students’ learning: a lesson in symmetry. Contemp. Educ. Psychol. 28, 187–204 (2003). https://doi.org/10.1016/S0361-476X(02)00007-3

    Article  Google Scholar 

  28. Yang, H., Meinel, C.: Content based lecture video retrieval using speech and video text information. IEEE Trans. Learn. Technol. 7, 142–154 (2014). https://doi.org/10.1109/TLT.2014.2307305

    Article  Google Scholar 

  29. Yang, M.T., Liao, W.C.: Computer-assisted culture learning in an online augmented reality environment based on free-hand gesture interaction. IEEE Trans. Learn. Technol. 7, 107–117 (2014). https://doi.org/10.1109/TLT.2014.2307297

    Article  Google Scholar 

  30. AL-Rousan, M., Assaleh, K., Tala’a, A.: Video-based signer-independent Arabic sign language recognition using hidden Markov models. Appl. Soft Comput. J. 9, 990–999 (2009). https://doi.org/10.1016/j.asoc.2009.01.002

  31. Chen, X., Wang, Z.J.: Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8, 184–192 (2013). https://doi.org/10.1016/j.bspc.2012.08.005

    Article  Google Scholar 

  32. Junker, H., Amft, O., Lukowicz, P., Tröster, G.: Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recogn. 41, 2010–2024 (2008). https://doi.org/10.1016/j.patcog.2007.11.016

    Article  MATH  Google Scholar 

  33. Hachaj, T., Ogiela, M.R.: Full body movements recognition-unsupervised learning approach with heuristic R-GDL method. Digit. Signal Process. A Rev. J. 46, 239–252 (2015). https://doi.org/10.1016/j.dsp.2015.07.004

    Article  MathSciNet  Google Scholar 

  34. Farella, E., Pieracci, A., Benini, L., Rocchi, L., Acquaviva, A.: Interfacing human and computer with wireless body area sensor networks: the WiMoCA solution. Multimed. Tools Appl. 38, 337–363 (2008). https://doi.org/10.1007/s11042-007-0189-5

    Article  Google Scholar 

  35. Johnson, L., Smith, R., Willis, H., Levine, A., Haywood, K.: The 2011 Horizon Report (2011)

    Google Scholar 

  36. Shih, C.H., Shih, C.T., Chu, C.L.: Assisting people with multiple disabilities actively correct abnormal standing posture with a Nintendo Wii Balance Board through controlling environmental stimulation. Res. Dev. Disabil. (2010). https://doi.org/10.1016/j.ridd.2010.03.004

    Article  Google Scholar 

  37. Shih, C.H., Shih, C.T., Chiang, M.S.: A new standing posture detector to enable people with multiple disabilities to control environmental stimulation by changing their standing posture through a commercial Wii Balance Board. Res. Dev. Disabil. (2010). https://doi.org/10.1016/j.ridd.2009.09.013

    Article  Google Scholar 

  38. Shih, C.H., Shih, C.J., Shih, C.T.: Assisting people with multiple disabilities by actively keeping the head in an upright position with a Nintendo Wii Remote Controller through the control of an environmental stimulation. Res. Dev. Disabil. (2011). https://doi.org/10.1016/j.ridd.2011.04.008

    Article  Google Scholar 

  39. Shih, C.H., Chung, C.C., Shih, C.T., Chen, L.C.: Enabling people with developmental disabilities to actively follow simple instructions and perform designated physical activities according to simple instructions with Nintendo Wii Balance Boards by controlling environmental stimulation. Res. Dev. Disabil. (2011). https://doi.org/10.1016/j.ridd.2011.05.031

    Article  Google Scholar 

  40. Shih, C.H., Chen, L.C., Shih, C.T.: Assisting people with disabilities to actively improve their collaborative physical activities with Nintendo Wii Balance Boards by controlling environmental stimulation. Res. Dev. Disabil. (2012). https://doi.org/10.1016/j.ridd.2011.08.006

    Article  Google Scholar 

  41. Shih, C.H., Chang, M.L., Mohua, Z.: A three-dimensional object orientation detector assisting people with developmental disabilities to control their environmental stimulation through simple occupational activities with a Nintendo Wii Remote Controller. Res. Dev. Disabil. (2012). https://doi.org/10.1016/j.ridd.2011.10.012

    Article  Google Scholar 

  42. Shih, C.H.: A standing location detector enabling people with developmental disabilities to control environmental stimulation through simple physical activities with Nintendo Wii Balance Boards. Res. Dev. Disabil. (2011). https://doi.org/10.1016/j.ridd.2010.11.011

    Article  Google Scholar 

  43. Shih, C.H., Chang, M.L.: A wireless object location detector enabling people with developmental disabilities to control environmental stimulation through simple occupational activities with Nintendo Wii Balance Boards. Res. Dev. Disabil. (2012). https://doi.org/10.1016/j.ridd.2011.12.018

    Article  Google Scholar 

  44. Nissenson, P.M., Shih, A.C.: MOOC on a budget: development and implementation of a low-cost MOOC at a state university. Comput. Educ. J. 7(1), 8 (2016)

    Google Scholar 

  45. Chan, J.C.P., Leung, H., Tang, J.K.T., Komura, T.: A virtual reality dance training system using motion capture technology. IEEE Trans. Learn. Technol. 4, 187–195 (2011). https://doi.org/10.1109/TLT.2010.27

    Article  Google Scholar 

  46. Hui-Mei, J.H.: The potential of kinect in education. Int. J. Inf. Educ. Technol. 1, 365–370 (2013). https://doi.org/10.7763/ijiet.2011.v1.59

    Article  Google Scholar 

  47. Schneider, J., Borner, D., Van Rosmalen, P., Specht, M.: Can you help me with my pitch? studying a tool for real-time automated feedback. IEEE Trans. Learn. Technol. 9, 318–327 (2016). https://doi.org/10.1109/TLT.2016.2627043

    Article  Google Scholar 

  48. Barakat, R.A.: Arabic gestures. J. Pop. Cult. (1973). https://doi.org/10.1111/j.0022-3840.1973.00749.x

    Article  Google Scholar 

  49. Enfield, N.J.: ‘Lip-pointing’: a discussion of form and function with reference to data from Laos. Gesture. 1, 185–211 (2002). https://doi.org/10.1075/gest.1.2.06enf

    Article  Google Scholar 

  50. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291–7299 (2017). https://doi.org/10.1109/CVPR.2017.143

  51. Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1145–1153 (2017). https://doi.org/10.1109/CVPR.2017.494

  52. Konrad, S.G., Shan, M., Masson, F.R., Worrall, S., Nebot, E.: Pedestrian Dynamic and Kinematic Information Obtained from Vision Sensors. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1299–1305. IEEE (2018). https://doi.org/10.1109/IVS.2018.8500527

  53. Copeland, M., Soh, J., Puca, A., Manning, M., Gollob, D.: Microsof  t azure machine learning. Microsoft Azure, pp. 355–380. Apress, Berkeley, CA (2015). https://doi.org/10.1007/978-1-4842-1043-7_14

    Chapter  Google Scholar 

  54. Caballero, D., et al.: ASR in classroom today: automatic visualization of conceptual network in science classrooms. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., Pérez-Sanagustín, M. (eds.) EC-TEL 2017. LNCS, vol. 10474, pp. 541–544. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66610-5_58

    Chapter  Google Scholar 

  55. Araya, R., Jiménez, A., Aguirre, C.: Context-based personalized predictors of the length of written responses to open-ended questions of elementary school students. In: Sieminski, A., Kozierkiewicz, A., Nunez, M., Ha, Q.T. (eds.) Modern Approaches for Intelligent Information and Database Systems. SCI, vol. 769, pp. 135–146. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-76081-0_12

    Chapter  Google Scholar 

Download references

Acknowledgements

Support from ANID/PIA/Basal Funds for Centers of Excellence FB0003, as well as FONDECYT 3170062 are gratefully acknowledged.

Author information

Authors and Affiliations

  1. Center for Advanced Research in Education, Institute of Education (IE) of Universidad de Chile, Santiago, Chile

    Josefina Hernández Correa, Danyal Farsani & Roberto Araya

Authors
  1. Josefina Hernández Correa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danyal Farsani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roberto Araya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Josefina Hernández Correa .

Editor information

Editors and Affiliations

  1. Wroclaw University of Economics and Business, Wrocław, Poland

    Marcin Hernes

  2. Wrocław University of Science and Technology, Wrocław, Poland

    Krystian Wojtkiewicz

  3. University of Newcastle, Newcastle, Australia

    Edward Szczerbicki

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hernández Correa, J., Farsani, D., Araya, R. (2020). An Application of Machine Learning and Image Processing to Automatically Detect Teachers’ Gestures. In: Hernes, M., Wojtkiewicz, K., Szczerbicki, E. (eds) Advances in Computational Collective Intelligence. ICCCI 2020. Communications in Computer and Information Science, vol 1287. Springer, Cham. https://doi.org/10.1007/978-3-030-63119-2_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-63119-2_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-63118-5

  • Online ISBN: 978-3-030-63119-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation