Abstract
Automatic recognition of facial expressions is a common problem in human-computer interaction. While humans can recognize facial expressions very easily, machines cannot do it as easily as humans. Analyzing facial changes during facial expressions is one of the methods used for this purpose by the machines. In this research, facial deformation caused by facial expressions is considered for automatic facial expression recognition by machines. To achieve this goal, the motion vectors of facial deformations are captured during facial expression using an optical flow algorithm. These motion vectors are then used to analyze facial expressions using some data mining algorithms. This analysis not only determined how changes in the face occur during facial expressions but can also be used for facial expression recognition. The facial expressions investigated in this research are happiness, sadness, surprise, fear, anger, and disgust. According to our research, these facial expressions were classified into 12 classes of facial motion vectors. We applied our proposed analysis mechanism to the extended Cohen-Kanade facial expression dataset. Our developed automatic facial expression system achieved 95.3%, 92.8%, and 90.2% accuracy using Deep Learning (DL), Support Vector Machine (SVM), and C5.0 classifiers, respectively. In addition, based on this research, it was determined which parts of the face have a greater impact on facial expression recognition.
S. Nahavandi—Associate Deputy Vice-Chancellor Research.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mehrabian, A.: Communication without words. In: Communication Theory, pp. 193–200. Routledge (2017)
Valstar, M.F., Mehu, M., Jiang, B., Pantic, M., Scherer, K.: Meta-analysis of the first facial expression recognition challenge. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42, 966–979 (2012)
Lisetti, C.L., Schiano, D.J.: Automatic facial expression interpretation: where human-computer interaction, artificial intelligence and cognitive science intersect. Pragmat. Cogn. 8, 185–235 (2000)
Sultan Zia, M., Hussain, M., Arfan Jaffar, M.: A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier. Multimed. Tools Appl. 77, 25537–25567 (2018)
Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1424–1445 (2000)
Bassili, J.N.: Facial motion in the perception of faces and of emotional expression. J. Exp. Psychol. Hum. Percept. Perform. 4, 373 (1978)
Vasanth, P., Nataraj, K.: Facial expression recognition using SVM classifier. Indon. J. Electr. Eng. Inform. (IJEEI) 3, 16–20 (2015)
Abdulrahman, M., Eleyan, A.: Facial expression recognition using support vector machines. In: 2015 23nd Signal Processing and Communications Applications Conference (SIU), pp. 276–279. IEEE (2015)
Xu, X., Quan, C., Ren, F.: Facial expression recognition based on Gabor wavelet transform and histogram of oriented gradients. In: 2015 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 2117–2122. IEEE (2015)
Naghsh-Nilchi, A.R., Roshanzamir, M.: An efficient algorithm for motion detection based facial expression recognition using optical flow. Proc. World Acad. Sci. Eng. Technol. 20, 23–28 (2006)
Dhavalikar, A.S., Kulkarni, R.K.: Face detection and facial expression recognition system. In: 2014 International Conference on Electronics and Communication Systems (ICECS), pp. 1–7 (2014)
Roshanzamir, M., Naghsh Nilchi, A.R., Roshanzamir, M.: A new fuzzy rule-based approach for automatic facial expression recognition. 1st National Conference on Soft Computing. Undefined (1394)
Liliana, D.Y., Basaruddin, T., Widyanto, M.R., Oriza, I.I.D.: Fuzzy emotion: a natural approach to automatic facial expression recognition from psychological perspective using fuzzy system. Cogn. Process. 20, 391–403 (2019)
Kirana, K.C., Wibawanto, S., Herwanto, H.W.: Facial emotion recognition based on viola-jones algorithm in the learning environment. In: 2018 International Seminar on Application for Technology of Information and Communication, pp. 406–410. IEEE (2018)
Happy, S., Routray, A.: Robust facial expression classification using shape and appearance features. In: 2015 Eighth International Conference on Advances in Pattern Recognition (ICAPR), pp. 1–5. IEEE (2015)
Chołoniewski, J., Chmiel, A., Sienkiewicz, J., Hołyst, J.A., Küster, D., Kappas, A.: Temporal Taylor’s scaling of facial electromyography and electrodermal activity in the course of emotional stimulation. Chaos Solitons Fractals 90, 91–100 (2016)
Tuncer, T., Dogan, S., Subasi, A.: A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos, Solitons Fractals 144, 110671 (2021)
Mehta, D., Siddiqui, M.F.H., Javaid, A.Y.: Recognition of emotion intensities using machine learning algorithms: a comparative study. Sensors 19, 1897 (2019)
Varma, S., Shinde, M., Chavan, S.S.: Analysis of PCA and LDA features for facial expression recognition using SVM and hmm classifiers. In: Pawar, P.M., Ronge, B.P., Balasubramaniam, R., Vibhute, A.S., Apte, S.S. (eds.) Techno-Societal 2018, pp. 109–119. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-16848-3_11
Islam, S.M.S., et al.: Machine learning approaches for predicting hypertension and its associated factors using population-level data from three south asian countries. Front. Cardiovasc. Med. 9, 839379 (2022)
Lu, Y., Wang, S., Zhao, W., Zhao, Y.: WGAN-based robust occluded facial expression recognition. IEEE Access 7, 93594–93610 (2019)
Nahavandi, D., Alizadehsani, R., Khosravi, A., Acharya, U.R.: Application of artificial intelligence in wearable devices: opportunities and challenges. Comput. Methods Programs Biomed. 213, 106541 (2022)
Abdellatif, D., El Moutaouakil, K., Satori, K.: Clustering and Jarque-Bera normality test to face recognition. Procedia Comput. Sci. 127, 246–255 (2018)
Malekzadeh, A., Zare, A., Yaghoobi, M., Alizadehsani, R.: Automatic diagnosis of epileptic seizures in EEG signals using fractal dimension features and convolutional autoencoder method. Big Data Cogn. Comput. 5, 78 (2021)
Abdulrazaq, M.B., Mahmood, M.R., Zeebaree, S.R., Abdulwahab, M.H., Zebari, R.R., Sallow, A.B.: An analytical appraisal for supervised classifiers’ performance on facial expression recognition based on relief-F feature selection. In: Journal of Physics: Conference Series, p. 012055. IOP Publishing (2021)
Barman, A., Dutta, P.: Facial expression recognition using distance and texture signature relevant features. Appl. Soft Comput. 77, 88–105 (2019)
Wang, C., Wang, S., Liang, G.: Identity-and pose-robust facial expression recognition through adversarial feature learning. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 238–246 (2019)
Malekzadeh, A., Zare, A., Yaghoobi, M., Kobravi, H.-R., Alizadehsani, R.: Epileptic seizures detection in EEG signals using fusion handcrafted and deep learning features. Sensors 21, 7710 (2021)
Saurav, S., Singh, S., Saini, R., Yadav, M.: Facial expression recognition using improved adaptive local ternary pattern. In: Chaudhuri, B.B., Nakagawa, M., Khanna, P., Kumar, S. (eds.) Proceedings of 3rd International Conference on Computer Vision and Image Processing. AISC, vol. 1024, pp. 39–52. Springer, Singapore (2020). https://doi.org/10.1007/978-981-32-9291-8_4
Rahul, M., Shukla, R., Goyal, P.K., Siddiqui, Z.A., Yadav, V.: Gabor filter and ICA-based facial expression recognition using two-layered hidden markov model. In: Gao, X.-Z., Tiwari, S., Trivedi, M.C., Mishra, K.K. (eds.) Advances in Computational Intelligence and Communication Technology. AISC, vol. 1086, pp. 511–518. Springer, Singapore (2021). https://doi.org/10.1007/978-981-15-1275-9_42
Rahul, M., Kohli, N., Agarwal, R.: Facial expression recognition using local multidirectional score pattern descriptor and modified hidden Markov model. Int. J. Adv. Intell. Paradig. 18, 538–551 (2021)
Gogić, I., Manhart, M., Pandžić, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. 36, 97–112 (2020)
Durmuşoğlu, A., Kahraman, Y.: Face expression recognition using a combination of local binary patterns and local phase quantization. In: 2021 International Conference on Communication, Control and Information Sciences (ICCISc), pp. 1–5. IEEE (2021)
Li, Y., Mavadati, S.M., Mahoor, M.H., Zhao, Y., Ji, Q.: Measuring the intensity of spontaneous facial action units with dynamic Bayesian network. Pattern Recogn. 48, 3417–3427 (2015)
Wang, L., Wang, K., Li, R.: Unsupervised feature selection based on spectral regression from manifold learning for facial expression recognition. IET Comput. Vision 9, 655–662 (2015)
Roshanzamir, M., et al.: Automatic facial expression recognition in an image sequence using conditional random field. In: 2022 IEEE 22nd International Symposium on Computational Intelligence and Informatics and 8th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (CINTI-MACRo), pp. 000271–000278 (2022)
Alizadehsani, R., et al.: Diagnosis of coronary artery disease using data mining based on lab data and echo features. J. Med. Bioeng. 1 (2012)
Alizadehsani, R., et al.: Diagnosis of coronary arteries stenosis using data mining. J. Med. Signals Sens. 2, 153 (2012)
Alizadehsani, R., Hosseini, M.J., Boghrati, R., Ghandeharioun, A., Khozeimeh, F., Sani, Z.A.: Exerting cost-sensitive and feature creation algorithms for coronary artery disease diagnosis. Int. J. Knowl. Discov. Bioinform. (IJKDB) 3, 59–79 (2012)
Sharifrazi, D., et al.: CNN-KCL: Automatic myocarditis diagnosis using convolutional neural network combined with k-means clustering (2020)
Asgharnezhad, H., et al.: Objective evaluation of deep uncertainty predictions for COVID-19 detection. Sci. Rep. 12, 815 (2022)
Roshanzamir, M., Alizadehsani, R., Roshanzamir, M., Shoeibi, A., Gorriz, J.M., Khosrave, A., Nahavandi, S.: What happens in Face during a facial expression? Using data mining techniques to analyze facial expression motion vectors. arXiv preprint arXiv:2109.05457 (2021)
Joloudari, J.H., et al.: DNN-GFE: a deep neural network model combined with global feature extractor for COVID-19 diagnosis based on CT scan images. EasyChair 2516–2314 (2021)
Ayoobi, N., et al.: Time series forecasting of new cases and new deaths rate for COVID-19 using deep learning methods. Results Phys. 27, 104495 (2021)
Javan, A.A.K., et al.: Medical images encryption based on adaptive-robust multi-mode synchronization of Chen hyper-chaotic systems. Sensors 21, 3925 (2021)
Qin, S., Zhu, Z., Zou, Y., Wang, X.: Facial expression recognition based on Gabor wavelet transform and 2-channel CNN. Int. J. Wavelets Multiresolut. Inf. Process. 18, 2050003 (2020)
Shoushtarian, M., et al.: Objective measurement of tinnitus using functional near-infrared spectroscopy and machine learning. PLoS One 15, e0241695 (2020)
Alizadehsani, R., et al.: Model uncertainty quantification for diagnosis of each main coronary artery stenosis. Soft. Comput. 24, 10149–10160 (2020)
Zangooei, M.H., Habibi, J., Alizadehsani, R.: Disease diagnosis with a hybrid method SVR using NSGA-II. Neurocomputing 136, 14–29 (2014)
Byun, S.-W., Lee, S.-P.: Human emotion recognition based on the weighted integration method using image sequences and acoustic features. Multimed. Tools Appl. 1–15 (2020)
Fernandez, P.D.M., Pena, F.A.G., Ren, T.I., Cunha, A.: FERAtt: facial expression recognition with attention net. arXiv preprint arXiv:1902.03284 3 (2019)
Alenazy, W.M., Alqahtani, A.S.: Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. J. Ambient. Intell. Humaniz. Comput. 12, 1631–1646 (2021)
Alexandre, G.R., Soares, J.M., Thé, G.A.P.: Systematic review of 3D facial expression recognition methods. Pattern Recogn. 100, 107108 (2020)
Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 13, 1195–1215 (2020)
Turan, C., Lam, K.-M.: Histogram-based local descriptors for facial expression recognition (FER): a comprehensive study. J. Vis. Commun. Image Represent. 55, 331–341 (2018)
Abdullah, S.M.S., Abdulazeez, A.M.: Facial expression recognition based on deep learning convolution neural network: a review. J. Soft Comput. Data Min. 2, 53–65 (2021)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (2010)
The SAGE Encyclopedia of Theory in Psychology. SAGE Publications, Inc., Thousand Oaks (2016)
Tan, P.-N., Steinbach, M., Kumar, V.: Introduction to Data Mining. Pearson Education India (2016)
Moravvej, S.V., et al.: RLMD-PA: a reinforcement learning-based myocarditis diagnosis combined with a population-based algorithm for pretraining weights. Contrast Media Mol. Imaging 2022 (2022)
Kiss, N., et al.: Comparison of the prevalence of 21 GLIM phenotypic and etiologic criteria combinations and association with 30-day outcomes in people with cancer: a retrospective observational study. Clin. Nutr. 41, 1102–1111 (2022)
Joloudari, J.H., et al.: Resource allocation optimization using artificial intelligence methods in various computing paradigms: a Review. arXiv preprint arXiv:2203.12315 (2022)
Alizadehsani, R., et al.: Factors associated with mortality in hospitalized cardiovascular disease patients infected with COVID-19. Immun. Inflamm. Dis. 10, e561 (2022)
Bishop, C.M.: Pattern recognition. Mach. Learn. 128 (2006)
Alizadehsani, R., et al.: Hybrid genetic-discretized algorithm to handle data uncertainty in diagnosing stenosis of coronary arteries. Expert. Syst. 39, e12573 (2022)
Khozeimeh, F., et al.: RF-CNN-F: random forest with convolutional neural network features for coronary artery disease diagnosis based on cardiac magnetic resonance. Sci. Rep. 12, 11178 (2022)
Kakhi, K., Alizadehsani, R., Kabir, H.D., Khosravi, A., Nahavandi, S., Acharya, U.R.: The internet of medical things and artificial intelligence: trends, challenges, and opportunities. Biocybern. Biomed. Eng. 42, 749–771 (2022)
Sharifrazi, D., et al.: Hypertrophic cardiomyopathy diagnosis based on cardiovascular magnetic resonance using deep learning techniques (2021)
Alizadehsani, R., et al.: Uncertainty-aware semi-supervised method using large unlabeled and limited labeled COVID-19 data. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 17, 1–24 (2021)
Shoeibi, A., et al.: Detection of epileptic seizures on EEG signals using ANFIS classifier, autoencoders and fuzzy entropies. Biomed. Signal Process. Control 73, 103417 (2022)
Hassannataj Joloudari, J., et al.: GSVMA: a genetic support vector machine ANOVA method for CAD diagnosis. Front. Cardiovasc. Med. 8, 760178 (2022)
Sonka, M., Hlavac, V., Boyle, R.: Image Processing, Analysis, and Machine Vision. Cengage Learning (2014)
Gautama, T., Hulle, M.A.V.: A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE Trans. Neural Netw. 13, 1127–1136 (2002)
Li, W., Hua, Y., Liangzheng, X.: Mouth detection based on interest point. In: 2007 Chinese Control Conference, pp. 610–613 (2007)
Bao, P.T., Nguyen, H., Nhan, D.: A new approach to mouth detection using neural network. In: 2009 IITA International Conference on Control, Automation and Systems Engineering (case 2009), pp. 616–619 (2009)
Wang, Q., Yang, S., Li, X.W.: A fast mouth detection algorithm based on face organs. In: 2009 2nd International Conference on Power Electronics and Intelligent Transportation System (PEITS), pp. 250–252 (2009)
Asadifard, M., Shanbezadeh, J.: Automatic adaptive center of pupil detection using face detection and cdf analysis. In: Proceedings of the International Multiconference of Engineers and Computer Scientists, p. 3. Citeseer (2010)
Bujlow, T., Riaz, T., Pedersen, J.M.: A method for classification of network traffic based on C5.0 machine learning algorithm. In: 2012 International Conference on Computing, Networking and Communications (ICNC), pp. 237–241 (2012)
Nahavandi, S., et al.: A Comprehensive Review on Autonomous Navigation. arXiv preprint arXiv:2212.12808 (2022)
Eskandarian, R., et al.: Identification of clinical features associated with mortality in COVID-19 patients. Oper. Res. Forum 4, 16 (2023)
Roshanzamir, M., et al.: Quantifying uncertainty in automated detection of Alzheimer’s patients using deep neural network (2023)
Iqbal, M.S., Ahmad, W., Alizadehsani, R., Hussain, S., Rehman, R.: Breast cancer dataset, classification and detection using deep learning. In: Healthcare, p. 2395. MDPI (2022)
Blömer, J., Otto, M., Seifert, J.-P.: A new CRT-RSA algorithm secure against bellcore attacks. In: Proceedings of the 10th ACM Conference on Computer and Communications Security, pp. 311–320. Association for Computing Machinery, Washington D.C. (2003)
Nasab, R.Z., et al.: Deep Learning in spatially resolved transcriptomics: a comprehensive technical view. arXiv preprint arXiv:2210.04453 (2022)
Abedini, S.S., et al.: A critical review of the impact of candidate copy number variants on autism spectrum disorders. arXiv preprint arXiv:2302.03211 (2023)
Danaei, S., et al.: Myocarditis diagnosis: a method using mutual learning-based abc and reinforcement learning. In: 2022 IEEE 22nd International Symposium on Computational Intelligence and Informatics and 8th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (CINTI-MACRo), pp. 000265–000270. IEEE (2022)
Bar-Itzhack, I.Y.: REQUEST-a recursive QUEST algorithm for sequential attitude determination. J. Guid. Control. Dyn. 19, 1034–1038 (1996)
Joloudari, J.H., et al.: Application of artificial intelligence techniques for automated detection of myocardial infarction: a review. Physiol. Meas. (2022)
Kabir, H., et al.: Uncertainty aware neural network from similarity and sensitivity. arXiv preprint arXiv:2304.14925 (2023)
Khalili, H., et al.: Prognosis prediction in traumatic brain injury patients using machine learning algorithms. Sci. Rep. 13, 960 (2023)
Lin, C.-L., Fan, C.-L.: Evaluation of CART, CHAID, and QUEST algorithms: a case study of construction defects in Taiwan. J. Asian Archit. Build. Eng. 18, 539–553 (2019)
Nematollahi, M.A., et al.: Association and predictive capability of body composition and diabetes mellitus using artificial intelligence: a cohort study (2022)
Abbasi Habashi, S., Koyuncu, M., Alizadehsani, R.: A survey of COVID-19 diagnosis using routine blood tests with the aid of artificial intelligence techniques. Diagnostics 13, 1749 (2023)
Karami, M., Alizadehsani, R., Argha, A., Dehzangi, I., Alinejad-Rokny, H.: Revolutionizing genomics with reinforcement learning techniques. arXiv preprint arXiv:2302.13268 (2023)
Khodatars, M., et al.: Deep learning for neuroimaging-based diagnosis and rehabilitation of autism spectrum disorder: a review. arXiv preprint arXiv:2007.01285 (2020)
Shoeibi, A., et al.: Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: a review. arXiv preprint arXiv:2105.04881 (2021)
Mahamivanan, H., et al.: Material recognition for construction quality monitoring using deep learning methods. Constr. Innov. (2023)
Khozeimeh, F., et al.: ALEC: active learning with ensemble of classifiers for clinical diagnosis of coronary artery disease. Comput. Biol. Med. 158, 106841 (2023)
Sadeghi, Z., et al.: A brief review of explainable artificial intelligence in healthcare. arXiv preprint arXiv:2304.01543 (2023)
Hong, L., et al.: GAN-LSTM-3D: An efficient method for lung tumour 3D reconstruction enhanced by attention-based LSTM. CAAI Trans. Intell. Technol. (2023)
Alizadehsani, R., et al.: A data mining approach for diagnosis of coronary artery disease. Comput. Methods Programs Biomed. 111, 52–61 (2013)
Alizadehsani, R., et al.: Swarm intelligence in internet of medical things: A review. Sensors 23, 1466 (2023)
Joloudari, J.H., et al.: BERT-deep CNN: state of the art for sentiment analysis of COVID-19 tweets. Soc. Netw. Anal. Min. 13, 99 (2023)
Nahavandi, D., Alizadehsani, R., Khosravi, A.: Integration of machine learning with wearable technologies. Handb. Hum.-Mach. Syst. 383–396 (2023)
Kiss, N., et al.: Machine learning models to predict outcomes at 30-days using Global Leadership Initiative on Malnutrition combinations with and without muscle mass in people with cancer. J. Cachexia Sarcopenia Muscle (2023)
Park, C.H., Park, H.: A comparison of generalized linear discriminant analysis algorithms. Pattern Recogn. 41, 1083–1097 (2008)
Nematollahi, M.A., et al.: Body composition predicts hypertension using machine learning methods: a cohort study. Sci. Rep. 13, 6885 (2023)
Khozeimeh, F., et al.: Importance of wearable health monitoring systems using IoMT; Requirements, advantages, disadvantages and challenges. In: 2022 IEEE 22nd International Symposium on Computational Intelligence and Informatics and 8th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (CINTI-MACRo), pp. 000245–000250. IEEE (2002)
Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1, 81–106 (1986)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Conflict of Interest
The authors have no competing interests to declare.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Roshanzamir, M. et al. (2024). Using Data Mining Techniques to Analyze Facial Expression Motion Vectors. In: Moosaei, H., HladÃk, M., Pardalos, P.M. (eds) Dynamics of Information Systems. DIS 2023. Lecture Notes in Computer Science, vol 14321. Springer, Cham. https://doi.org/10.1007/978-3-031-50320-7_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-50320-7_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50319-1
Online ISBN: 978-3-031-50320-7
eBook Packages: Computer ScienceComputer Science (R0)