Skip to main content
SpringerLink
Log in

2-D canonical correlation analysis based image super-resolution scheme for facial emotion recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this research work, a new Image super-resolution-based Face Emotion Recognition Model has been introduced. The proposed work includes two major phases: (a) Facial image super-resolution and (b) Facial emotion recognition. Initially, the collected facial image is subjected to the facial image super-resolution phase, where the Higher Resolution (HR) facial images are subjected to two-dimensional canonical correlation analysis (2D CCA). The acquired HR facial images are considered as the input for facial emotion recognition. From the acquired HR facial images, the face region alone (lips, eyes, and cheeks) is detected by Viola-Jones facial detection model. Subsequently, from the acquired facial regions, the most relevant features like proposed “Geometric Mean based Weighted Local Binary Pattern (GM-WLBP), Gray Level Co-occurrence Matrix (GLCM)” and generalized low-rank model (GLRM) features are extracted. Then, Principal Component Analysis (PCA) technique is applied to solve the curse of dimensionality. Finally, the reduced dimensional features are given to the emotion classification phase to classify the emotions as sad, happy, fear, rage, disgust, and surprise. The proposed hybrid classifier framework includes the renowned Long-Short Term Memory Network (LSTM) and Convolutional Neural Network (CNN) models. These deep learning models is separately trained using the dimensionally reduced features, and the outcomes are combined. Then, the mean value is computed on the final combined outcome (output of LSTM+ output of CNN), which results in the type of emotions. To enhance the classification accuracy, the weight of CNN is fine-tuned by a new Improved Tunicate swarm Optimization Model (ITSA), which is the conceptual improvement of standard Tunicate swarm Optimization (TSA). The performance of the proposed work is evaluated over the existing model to show the supremacy of the proposed work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Amiri M, Ahmadyfard A, Abolghasemi V (2019) A fast video super resolution for facial image. Signal Process Image Commun

  2. An L, Bhanu B (2014) Face image super-resolution using 2D CCA. Signal Process 103

  3. Anita JS, Abinaya JS (2019) Impact of supervised classifier on speech emotion recognition. Multimed Res 2(1):9–16

    Google Scholar 

  4. Cai J, Han H, Shan S, Chen X (2020) FCSR-GAN: joint face completion and super-resolution via multi-task learning. IEEE Trans Biom Behav Identity Sci 2(2):109–121. https://doi.org/10.1109/TBIOM.2019.2951063

    Article  Google Scholar 

  5. Cao L, Liu J, Du K, Guo Y, Wang T (2020) Guided cascaded super-resolution network for face image. IEEE Access 8:173387–173400. https://doi.org/10.1109/ACCESS.2020.3025972

    Article  Google Scholar 

  6. Chen L, Pan J, Li Q (2019) Robust face image super-resolution via joint learning of subdivided contextual model. IEEE Trans Image Process 28(12):5897–5909. https://doi.org/10.1109/TIP.2019.2920510

    Article  MathSciNet  MATH  Google Scholar 

  7. Chen J, Chen J, Wang Z, Liang C, Lin C-W (2020) Identity-aware face super-resolution for low-resolution face recognition. IEEE Signal Process Lett 27:645–649. https://doi.org/10.1109/LSP.2020.2986942

    Article  Google Scholar 

  8. Chen L, Pan J, Jiang J, Zhang J, Wu Y (2020) Robust face super-resolution via position relation model based on global face context. IEEE Trans Image Process 29:9002–9016. https://doi.org/10.1109/TIP.2020.3023580

    Article  MathSciNet  Google Scholar 

  9. Chen L, Pan J, Hu R, Han Z, Liang C, Wu Y (Dec. 2020) Modeling and optimizing of the multi-layer nearest neighbor network for face image super-resolution. IEEE Trans Circuits Syst Video Technol 30(12):4513–4525. https://doi.org/10.1109/TCSVT.2019.2917511

    Article  Google Scholar 

  10. Chen C, Gong D, Wang H, Li Z, Wong K-YK (2021) Learning spatial attention for face super-resolution. IEEE Trans Image Process 30:1219–1231. https://doi.org/10.1109/TIP.2020.3043093

    Article  Google Scholar 

  11. Chen K, Franko K, Sang R (2021) Structured model pruning of convolutional networks on tensor processing units, arXiv preprint arXiv:2107.04191

  12. Darekar RV, Dhande AP (2019) Emotion recognition from speech signals using DCNN with hybrid GA-GWO algorithm. Multimed Res 2(4):12–22

    Google Scholar 

  13. Farrugia RA, Guillemot C (Sept. 2017) Face hallucination using linear models of coupled sparse support. IEEE Trans Image Process 26(9):4562–4577. https://doi.org/10.1109/TIP.2017.2717181

    Article  MathSciNet  MATH  Google Scholar 

  14. Grm K, Scheirer WJ, Štruc V (2020) Face hallucination using cascaded super-resolution and identity priors. IEEE Trans Image Process 29:2150–2165. https://doi.org/10.1109/TIP.2019.2945835

    Article  Google Scholar 

  15. Hu X, Fan Z, Xuan Z (2021) Towards effective learning for face super-resolution with shape and pose perturbations. Knowl-Based Syst

  16. Ismail M, Anjum G Reddy TB (2020) Variable block size hybrid fractal technique for image compression. Proceedings IEEE 6th International Conference on Advanced Computing & Communication Systems, pp 510–515

  17. Jiang J, Ma J, Chen C, Jiang X, Wang Z (2017) Noise robust face image super-resolution through smooth sparse representation. IEEE Trans Cybern 47(11):3991–4002. https://doi.org/10.1109/TCYB.2016.2594184

    Article  Google Scholar 

  18. Jiang J, Chen C, Ma J, Wang Z, Wang Z, Hu R (2017) SRLSP: a face image super-resolution algorithm using smooth regression with local structure prior. IEEE Trans Multimed 19(1):27–40. https://doi.org/10.1109/TMM.2016.2601020

    Article  Google Scholar 

  19. Kaura S, Awasthia LK, Sangala AL, Dhimanb G (2020) Tunicate Swarm Algorithm: a new bio-inspired based metaheuristic paradigm for global optimization. Eng Appl Artif Intell 90

  20. Koulierakis I, Siolas G, Efthimiou E, Fotinea E, Stafylopatis A-G (2020) Recognition of static features in sign language using key-points. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pp 123–126

  21. Kumar Gola K, Chaurasia N, Gupta B, Singh Niranjan D (2021) Sea lion optimization algorithm based node deployment strategy in underwater acoustic sensor network. Int J Commun Syst 34(5):e4723

    Article  Google Scholar 

  22. Li J, Zhou Y, Ding J, Chen C, Yang X (2020) ID preserving face super-resolution generative adversarial networks. IEEE Access 8:138373–138381. https://doi.org/10.1109/ACCESS.2020.3011699

    Article  Google Scholar 

  23. Li M, Zhang Z, Yu J, Chen CW (2021) Learning face image super-resolution through facial semantic attribute transformation and self-attentive structure enhancement. IEEE Trans Multimed 23:468–483. https://doi.org/10.1109/TMM.2020.2984092

    Article  Google Scholar 

  24. Liu B, Ait-Boudaoud D (2019) Effective image super resolution via hierarchical convolutional neural network. Neurocomputing

  25. Liu Z-S, Siu W-C, Chan Y-L (2019) Reference based face super-resolution. IEEE Access 7:129112–129126. https://doi.org/10.1109/ACCESS.2019.2934078

    Article  Google Scholar 

  26. Liu L, Wang S, Wan L (2019) Component semantic prior guided generative adversarial network for face super-resolution. IEEE Access 7:77027–77036. https://doi.org/10.1109/ACCESS.2019.2921859

    Article  Google Scholar 

  27. Liu Q, Jia R, Zhao C, Liu X, Sun H, Zhang X (2020) Face super-resolution reconstruction based on self-attention residual network. IEEE Access 8:4110–4121. https://doi.org/10.1109/ACCESS.2019.2962790

    Article  Google Scholar 

  28. Lu T, Xiong Z, Zhang Y, Wang B, Lu T (2017) Robust face super-resolution via locality-constrained low-rank representation. IEEE Access 5:13103–13117. https://doi.org/10.1109/ACCESS.2017.2717963

    Article  Google Scholar 

  29. Lu T, Chen X, Zhang Y, Chen C, Xiong Z (2018) SLR: semi-coupled locality constrained representation for very low resolution face recognition and super resolution. IEEE Access 6:56269–56281. https://doi.org/10.1109/ACCESS.2018.2872761

    Article  Google Scholar 

  30. Lu T, Wang J, Zhang Y (2020) Global-local fusion network for face super-resolution. Neurocomputing 387:309–320

    Article  Google Scholar 

  31. Mohammad I, Harsha Vardhan V, Aditya Mounika V, Padmini KS (2019) An effective heart disease prediction method using artificial neural network. Int J Innov Technol Explor Eng 8(8):1529–1532

    Google Scholar 

  32. Mohammed Ismail B, Shaik MB, Reddy BE (2012) High rate compression based on luminance & chrominance of the image using binary plane technique. J Theor Appl Inf Technol 42(2):191–195

    Google Scholar 

  33. Mohammed Ismail B, Shaik MB, Reddy BE (2015) Improved fractal image compression using range block size. Proceedings of IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), pp 284–289

  34. Mohammed Ismail B, Reddy TB, Reddy BE (2016) Spiral architecture based hybrid fractal image compression. IEEE 2016 International Conference on Electrical, Electronics, Communication, Computer and Optimization Techniques (ICEECCOT)

  35. Mohammed Ismail B, Rajesh P, Alam M (2020) A machine learning based improved logisticregression method for prostate cancer diagnosis. Int J Emerg Trends Eng Res 8(9):5693–5698

    Article  Google Scholar 

  36. Mohammed Ismail B, Alam M, Tahernezhadi M, Vege HK, Rajesh P (2020) A machine learning classification technique for predicting prostate cancer. 2020 IEEE International Conference on Electro Information Technology (EIT) July 2020, pp 228–232

  37. Nagar S, Jain A, Kumar A (2020) Mixed-noise robust face super-resolution through residual-learning based error suppressed nearest neighbor representation. Inf Sci 546:121–145

    Article  MathSciNet  Google Scholar 

  38. Pei X, Dong T, Guan Y (2019) Super-resolution of face images using weighted elastic net constrained sparse representation. IEEE Access 7:55180–55190. https://doi.org/10.1109/ACCESS.2019.2913008

    Article  Google Scholar 

  39. Potamias R-A, Siolas G, Stafylopatis A (2019) A robust deep ensemble classifier for figurative language detection. In: International conference on engineering applications of neural networks. Springer, Cham, pp 164–175

  40. Rajakumar B R (2018) Optimization using lion algorithm: a biological inspiration from lion’s social behavior. Evolutionary Intelligence, Special Issue on Nature inspired algorithms for high performance computing in computer vision, Vol. 11, No. 1–2, pages 31–52, DOI: https://doi.org/10.1007/s12065-018-0168-y

  41. Sarkar A (2020) Optimization assisted convolutional neural network for facial emotion recognition. Multimed Res 3(2)

  42. Shahne R, Ismail M, Prabhu CSR (2019) Survey on deep learning techniques for prognosis and diagnosis ofCancer from microarray gene expression data. J Comput Theor Nanosci 16(12):5078–5088

    Article  Google Scholar 

  43. Sharma S Self supervised methods towards human activity recognition. IOSR Journal of Computer Engineering (IOSR-JCE) 22(6):51–56

  44. Shi J, Liu X, Zong Y, Qi C, Zhao G (June 2018) Hallucinating face image by regularization models in high-resolution feature space. IEEE Trans Image Process 27(6):2980–2995. https://doi.org/10.1109/TIP.2018.2813163

    Article  MathSciNet  MATH  Google Scholar 

  45. Wang H, Hu Q, Wu H (2021) DCLNet: dual closed-loop networks for face super-resolution. Knowl-Based Syst 222:106987

    Article  Google Scholar 

  46. Yan Y, Zhang Z, Wang H (2020) Low-resolution facial expression recognition: a filter learning perspective. Signal Process 169:107370

    Article  Google Scholar 

  47. Yang S, Liu J, Fang Y, Guo Z (2018) Joint-feature guided depth map super-resolution with face priors. IEEE Trans Cybern 48(1):399–411. https://doi.org/10.1109/TCYB.2016.2638856

    Article  Google Scholar 

  48. Yu X, Porikli F (2018) Imagining the unimaginable faces by Deconvolutional networks. IEEE Trans Image Process 27(6):2747–2761. https://doi.org/10.1109/TIP.2018.2808840

    Article  MathSciNet  MATH  Google Scholar 

  49. Yu X, Fernando B, Hartley R, Porikli F (2020) Semantic face hallucination: super-resolving very low-resolution face images with supplementary attributes. IEEE Trans Pattern Anal Mach Intell 42(11):2926–2943. https://doi.org/10.1109/TPAMI.2019.2916881

    Article  Google Scholar 

  50. Yuan Y-H, Li J (2021) Furong Peng," OPLS-SR: a novel face super-resolution learning method using orthonormalized coherent features". Inf Sci 561:52–69

    Article  Google Scholar 

  51. Yun JU, Jo B, Park IK (2020) Joint face super-resolution and Deblurring using generative adversarial network. IEEE Access 8:159661–159671. https://doi.org/10.1109/ACCESS.2020.3020729

    Article  Google Scholar 

  52. Zhang Y, Tsang IW, Li J, Liu P, Lu X, Yu X (2021) Face hallucination with finishing touches. IEEE Trans Image Process 30:1728–1743. https://doi.org/10.1109/TIP.2020.3046918

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Zhengzhou University, Zhengzhou, China

    Zia ullah & Lin Qi

  2. Resbee Info technologies Private Limited, Thuckalay, Tamil Nadu, India

    D. Binu & B. R. Rajakumar

  3. Department of Information Technology, Kannur University campus, Mangattuparamba, Kerala, India

    B. Mohammed Ismail

Authors
  1. Zia ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lin Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Binu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. B. R. Rajakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. B. Mohammed Ismail
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zia ullah.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

ullah, Z., Qi, L., Binu, D. et al. 2-D canonical correlation analysis based image super-resolution scheme for facial emotion recognition. Multimed Tools Appl 81, 13911–13934 (2022). https://doi.org/10.1007/s11042-022-11922-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-11922-3

Keywords

Search

Navigation