Abstract
Gesture recognition allows humans to interface and interact naturally with the machine. This paper presents analytical and algebraic methods to recognize specific combinations of facial expressions and hand gestures, including interactions between hands and face. The methodologies for extracting the features for both faces and hands were implemented starting from landmarks identified in real-time by the MediaPipe framework. To benchmark our approach, we selected a large set of emoji and designed a system capable of associating chosen emoji to facial expressions and/or hand gestures recognized. Complex poses and gestures combinations have been selected and assigned to specific emoji to be recognized by the system. Furthermore, the Web Application we created demonstrates that our system is able to quickly recognize facial expressions and complex poses from a video sequence from standard camera. The experimental results show that our proposed methods are generalizable, robust and achieve on average 99,25% of recognition accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
\(max_{x}\) = max{1.x, 5.x, 9.x, 13.x, 17.x}, \(min_{x}\) = min{1.x, 5.x, 9.x, 13.x, 17.x}, \(max_{y}\) = max{1.y, 5.y, 9.y, 13.y, 17.y}, \(min_{y}\) = min{1.y, 5.y, 9.y, 13.y, 17.y}, Max = max{0.y, 1.y, 5.y, 9.y, 13.y, 17.y}, Min = min{0.y, 1.y, 5.y, 9.y, 13.y, 17.y}.
- 2.
\(max_{x}\) = max{mcp.x, pip.x, dip.x} \(+\) offset, \(min_{x}\) = min{mcp.x, pip.x, dip.x} − offset,
\(max_{y}\) = max{mcp.y, pip.y, dip.y} \(+\) offest, \(min_{y}\) = min{mcp.y, pip.y, dip.y} − offset.
- 3.
\(max_{x}\) = max{mcp.x, pip.x} \(+\) offest, \(min_{x}\) = min{mcp.x, pip.x} − offset,
\(max_{y}\) = max{mcp.y, pip.y} \(+\) offset, \(min_{y}\) = min{mcp.y, pip.y} − offest.
- 4.
\(max_{x}\) = {mcp.x} \(+\) offset2, \(\min _{x}\) = {mcp.x} − offset2, \(max_{y}\) = {mcp.y} \(+\) offest2, \(min_{y}\) = {mcp.y} − offset2.
- 5.
For further details refer to the complete set of encoded emojis and related gestures reported in the supplementary material available at the following link.
- 6.
The video related to a complete test is available on the supplementary material.
References
Clough, S., Duff, M.C.: The role of gesture in communication and cognition: implications for understanding and treating neurogenic communication disorders. Front. Hum. Neurosci. 14, 1–22 (2020)
Battiato, S., Conoci, S., Leotta, R., Ortis, A., Rundo, F., Trenta, F.: Benchmarking of computer vision algorithms for driver monitoring on automotive-grade devices. In 2020 AEIT International Conference of Electrical and Electronic Technologies for Automotive (AEIT AUTOMOTIVE), pp. 1–6. IEEE (November 2020)
Altameem, T., Altameem, A.: Facial expression recognition using human machine interaction and multi-modal visualization analysis for healthcare applications. Image Vis. Comput. 103, 104044 (2020)
Dey, S., Laha, A., Paul, A., Roy, S., Paul, S.: Facial expression recognition in video call. Int. J. Eng. Res. Technol. (IJERT) 09(11), 159–161 (2021). NCETER - 2021
Munasinghe, M.I.N.P.: Facial expression recognition using facial landmarks and random forest classifier. In 2018 IEEE/ACIS 17th International Conference on Computer and Information Science (ICIS), pp. 423–427. IEEE (June 2018)
Rohith Raj, S., Pratiba, D., Ramakanth Kumar, P.: Facial expression recognition using facial landmarks: a novel approach. ASETS J. 5, 24–28 (2020)
Wang, M., Tan, P., Zhang, X., Kang, Yu., Jin, C., Cao, J.: Facial expression recognition based on CNN. J. Phys: Conf. Ser. 1601(5), 052027 (2020)
Matos, A., Filipe, V., Couto, P.: Human-computer interaction based on facial expression recognition: a case study in degenerative neuromuscular disease. In: Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, pp. 8–12 (December 2016)
Borghi, G., Frigieri, E., Vezzani, R., Cucchiara, R.: Hands on the wheel: a dataset for driver hand detection and tracking. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, pp. 564–570. IEEE (May 2018)
Gupta, S., Jaafar, J., Ahmad, W.F.W.: Static hand gesture recognition using local Gabor filter. Procedia Eng. 41, 827–832 (2012)
Chen, F.S., Fu, C.M., Huang, C.L.: Hand gesture recognition using a real-time tracking method and hidden Markov models. Image Vis. Comput. 21(8), 745–758 (2003)
Koh, J.I.: Developing a hand gesture recognition system for mapping symbolic hand gestures to analogous emoji in computer-mediated communications. In: Proceedings of the 25th International Conference on Intelligent User Interfaces Companion (2020)
Song, N., Yang, H., Wu, P.: A gesture-to-emotional speech conversion by combining gesture recognition and facial expression recognition. In 2018 1st Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), pp. 1–6. IEEE (May 2018)
Liu, N., Lovell, B.C.: Hand gesture extraction by active shape models. In: Digital Image Computing: Techniques and Applications, DICTA 2005, p. 10. IEEE (December 2005)
Elmezain, M., Al-Hamadi, A., Pathan, S.S., Michaelis, B.: Spatio-temporal feature extraction-based hand gesture recognition for isolated American sign language and Arabic numbers. In: 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis, pp. 254–259. IEEE (September 2009)
Kartynnik, Y., Ablavatski, A., Grishchenko, I., Grundmann, M.: Real-time facial surface geometry from monocular video on mobile GPUs. arXiv preprint arXiv:1907.06724 (2019)
Zhang, F., et al.: MediaPipe hands: on-device real-time hand tracking. arXiv preprint arXiv:2006.10214 (2020)
Ortis, A., Farinella, G.M., Battiato, S.: Survey on visual sentiment analysis. IET Image Process. 14(8), 1440–1456 (2020)
Soukupova, T., Cech, J.: Eye blink detection using facial landmarks. In: 21st Computer Vision Winter Workshop, Rimske Toplice, Slovenia (February 2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zuccarà, R., Ortis, A., Battiato, S. (2022). Recognition of Complex Gestures for Real-Time Emoji Assignment. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13374. Springer, Cham. https://doi.org/10.1007/978-3-031-13324-4_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-13324-4_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-13323-7
Online ISBN: 978-3-031-13324-4
eBook Packages: Computer ScienceComputer Science (R0)