Skip to main content
SpringerLink
Log in

A Privacy-Preserving Federated-MobileNet for Facial Expression Detection from Images

  • Conference paper
Applied Intelligence and Informatics (AII 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1724))

Included in the following conference series:

Abstract

Facial expression recognition is an intriguing research area that has been explored and utilized in a wide range of applications such as health, security, and human-computer interactions. The ability to recognize facial expressions accurately is crucial for human-computer interactions. However, most of the facial expression analysis techniques have so far paid little or no concern to users’ data privacy. To overcome this concern, in this paper, we incorporated Federated Learning (FL) as a privacy-preserving machine learning approach in the field of facial expression recognition to develop a shared model without exposing personal information. The individual models are trained on the different client devices where the data is stored. In this work, a lightweight Convolutional Neural Network (CNN) model called the MobileNet architecture is utilised to detect expressions from facial images. To evaluate the model, two publicly available datasets are used and several experiments are conducted. The result shows that the proposed privacy-preserving Federated-MobileNet approach could recognize facial expressions with considerable accuracy compared to the general approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Code Availability

The code and datasets are available at https://github.com/tapu1996/AII-2022-Federated-Learning-96104.

References

  1. Ahmed, S., Hossain, M.F., Nur, S.B., Shamim Kaiser, M., Mahmud, M.: Toward machine learning-based psychological assessment of autism spectrum disorders in school and community. In: Kaiser, M.S., Bandyopadhyay, A., Ray, K., Singh, R., Nagar, V. (eds.) Proceedings of Trends in Electronics and Health Informatics. LNNS, vol. 376, pp. 139–149. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-8826-3_13

    Chapter  Google Scholar 

  2. Niamat Ullah Akhund, T.M., Mahi, M.J.N., Hasnat Tanvir, A.N.M., Mahmud, M., Kaiser, M.S.: ADEPTNESS: Alzheimer’s disease patient management system using pervasive sensors - early prototype and preliminary results. In: Wang, S., et al. (eds.) BI 2018. LNCS (LNAI), vol. 11309, pp. 413–422. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05587-5_39

    Chapter  Google Scholar 

  3. Akter, T., Ali, M.H., Satu, M.S., Khan, M.I., Mahmud, M.: Towards autism subtype detection through identification of discriminatory factors using machine learning. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 401–410. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_36

    Chapter  Google Scholar 

  4. Al Banna, M.H., Ghosh, T., Taher, K.A., Kaiser, M.S., Mahmud, M.: A monitoring system for patients of autism spectrum disorder using artificial intelligence. In: Mahmud, M., Vassanelli, S., Kaiser, M.S., Zhong, N. (eds.) BI 2020. LNCS (LNAI), vol. 12241, pp. 251–262. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59277-6_23

    Chapter  Google Scholar 

  5. Al Mamun, S., Kaiser, M.S., Mahmud, M.: An artificial intelligence based approach towards inclusive healthcare provisioning in society 5.0: a perspective on brain disorder. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 157–169. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_15

    Chapter  Google Scholar 

  6. Biswas, M., Kaiser, M.S., Mahmud, M., Al Mamun, S., Hossain, M.S., Rahman, M.A.: An XAI based autism detection: the context behind the detection. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 448–459. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_40

    Chapter  Google Scholar 

  7. Biswas, M., et al.: Indoor navigation support system for patients with neurodegenerative diseases. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 411–422. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_37

    Chapter  Google Scholar 

  8. Chatterjee, A., Gupta, U., Chinnakotla, M.K., Srikanth, R., Galley, M., Agrawal, P.: Understanding emotions in text using deep learning and big data. Comput. Hum. Behav. 93, 309–317 (2019)

    Article  Google Scholar 

  9. Chen, J., Hu, B., Moore, P., Zhang, X., Ma, X.: Electroencephalogram-based emotion assessment system using ontology and data mining techniques. Appl. Soft Comput. 30, 663–674 (2015)

    Article  Google Scholar 

  10. Chhikara, P., Singh, P., Tekchandani, R., Kumar, N., Guizani, M.: Federated learning meets human emotions: a decentralized framework for human-computer interaction for IoT applications. IEEE Internet Things J. 8(8), 6949–6962 (2020)

    Article  Google Scholar 

  11. Courville, P., Goodfellow, A., Mirza, I., Bengio, Y.: Fer-2013 face database. Universit de Montreal, Montréal (2013)

    Google Scholar 

  12. Gharaee, Z., Gärdenfors, P., Johnsson, M.: First and second order dynamics in a hierarchical SOM system for action recognition. Appl. Soft Comput. 59, 574–585 (2017)

    Article  Google Scholar 

  13. Ghosh, T., et al.: An attention-based mood controlling framework for social media users. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 245–256. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_23

    Chapter  Google Scholar 

  14. Ghosh, T., et al.: Artificial intelligence and internet of things in screening and management of autism spectrum disorder. Sustain. Cities Soc. 74, 103189 (2021)

    Article  Google Scholar 

  15. Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  16. Huang, K.Y., Wu, C.H., Su, M.H., Kuo, Y.T.: Detecting unipolar and bipolar depressive disorders from elicited speech responses using latent affective structure model. IEEE Trans. Affect. Comput. 11(3), 393–404 (2018)

    Article  Google Scholar 

  17. Huang, Z., Epps, J., Joachim, D.: Speech landmark bigrams for depression detection from naturalistic smartphone speech. In: Proceedings of ICASSP, pp. 5856–5860 (2019)

    Google Scholar 

  18. Jain, D.K., Shamsolmoali, P., Sehdev, P.: Extended deep neural network for facial emotion recognition. Pattern Recognit. Lett. 120, 69–74 (2019)

    Article  Google Scholar 

  19. Jesmin, S., Kaiser, M.S., Mahmud, M.: Towards artificial intelligence driven stress monitoring for mental wellbeing tracking during COVID-19. In: Proceedings of WI-IAT, pp. 845–851 (2020)

    Google Scholar 

  20. Khaireddin, Y., Chen, Z.: Facial emotion recognition: state of the art performance on fer2013. arXiv preprint arXiv:2105.03588 (2021)

  21. Kulkarni, S.S., Reddy, N.P., Hariharan, S.: Facial expression (mood) recognition from facial images using committee neural networks. Biomed. Eng. Online 8(1), 1–12 (2009)

    Article  Google Scholar 

  22. Kwon, S., et al.: Att-Net: enhanced emotion recognition system using lightweight self-attention module. Appl. Soft Comput. 102, 107101 (2021)

    Article  Google Scholar 

  23. Latif, S., Asim, M., Rana, R., Khalifa, S., Jurdak, R., Schuller, B.W.: Augmenting generative adversarial networks for speech emotion recognition. arXiv preprint arXiv:2005.08447 (2020)

  24. Lee, H.C., Wu, C.Y., Lin, T.M.: Facial expression recognition using image processing techniques and neural networks. In: Pan, J.S., Yang, C.N., Lin, C.C. (eds.) Advances in Intelligent Systems and Applications. Smart Innovation, Systems and Technologies, vol. 21, pp. 259–267. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35473-1_26

    Chapter  Google Scholar 

  25. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: Proceeedings of IEEE CVPR, pp. 94–101 (2010)

    Google Scholar 

  26. Mahmud, M., et al.: Towards explainable and privacy-preserving artificial intelligence for personalisation in autism spectrum disorder. In: Antona, M., Stephanidis, C. (eds.) HCII 2022. LNCS, vol. 13309, pp. 356–370. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05039-8_26

    Chapter  Google Scholar 

  27. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics , pp. 1273–1282 (2017)

    Google Scholar 

  28. Mehendale, N.: Facial emotion recognition using convolutional neural networks (FERC). SN Appl. Sci. 2(3), 1–8 (2020)

    Article  Google Scholar 

  29. Merler, M., et al.: Automatic curation of sports highlights using multimodal excitement features. IEEE Trans. Multimedia 21(5), 1147–1160 (2018)

    Article  Google Scholar 

  30. Al Nahian, M.J., Ghosh, T., Uddin, M.N., Islam, M.M., Mahmud, M., Kaiser, M.S.: Towards artificial intelligence driven emotion aware fall monitoring framework suitable for elderly people with neurological disorder. In: Mahmud, M., Vassanelli, S., Kaiser, M.S., Zhong, N. (eds.) BI 2020. LNCS (LNAI), vol. 12241, pp. 275–286. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59277-6_25

    Chapter  Google Scholar 

  31. Nan, Y., Ju, J., Hua, Q., Zhang, H., Wang, B.: A-mobilenet: an approach of facial expression recognition. Alex. Eng. J. 61(6), 4435–4444 (2022)

    Article  Google Scholar 

  32. Rana, R., Latif, S., Gururajan, R., Gray, A., Mackenzie, G., Humphris, G., Dunn, J.: Automated screening for distress: a perspective for the future. Eur. J. Cancer Care 28(4), e13033 (2019)

    Article  Google Scholar 

  33. Schoneveld, L., Othmani, A., Abdelkawy, H.: Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recognit. Lett. 146, 1–7 (2021)

    Article  Google Scholar 

  34. Shaffi, N., et al.: Triplet-loss based siamese convolutional neural network for 4-way classification of alzheimerÕs disease. In: Mahmud, M., He, J., Vassanelli, S., van Zundert, A., Zhong, N. (eds.) BI. LNCS, vol. 13406, pp. 277–287. Springer, Cham (2022)

    Chapter  Google Scholar 

  35. Smith, V., Chiang, C.K., Sanjabi, M., Talwalkar, A.S.: Federated multi-task learning. In: Advance in Neural Information Processing System, vol. 30 (2017)

    Google Scholar 

  36. Sumi, A.I., Zohora, M.F., Mahjabeen, M., Faria, T.J., Mahmud, M., Kaiser, M.S.: fASSERT: A fuzzy assistive system for children with autism using internet of things. In: Wang, S., et al. (eds.) BI 2018. LNCS (LNAI), vol. 11309, pp. 403–412. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05587-5_38

    Chapter  Google Scholar 

  37. Tsouvalas, V., Ozcelebi, T., Meratnia, N.: Privacy-preserving speech emotion recognition through semi-supervised federated learning. arXiv preprint arXiv:2202.02611 (2022)

  38. Vögel, H.J., et al.: Emotion-awareness for intelligent vehicle assistants: A research agenda. In: Proceedings of SEFAIAS, pp. 11–15 (2018)

    Google Scholar 

  39. Wadhera, T., Mahmud, M.: Brain networks in autism spectrum disorder, epilepsy and their relationship: a machine learning approach. In: Chen, T., Carter, J., Mahmud, M., Khuman, A.S. (eds.) Artificial Intelligence in Healthcare. Brain Informatics and Health, pp. 125–142. Springer, Singapore (2022). https://doi.org/10.1007/978-981-19-5272-2_6

    Chapter  Google Scholar 

  40. Wadhera, T., Mahmud, M.: Influences of social learning in individual perception and decision making in people with autism: a computational approach. In: Mahmud, M., He, J., Vassanelli, S., van Zundert, A., Zhong, N. (eds.) BI 2022. LNCS, vol. 13406, pp. 50–61. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-15037-1_5

    Chapter  Google Scholar 

  41. Yang, J., Liu, J., Han, R., Wu, J.: Transferable face image privacy protection based on federated learning and ensemble models. Complex Intell. Syst. 7(5), 2299–2315 (2021). https://doi.org/10.1007/s40747-021-00399-6

    Article  Google Scholar 

  42. Yurochkin, M., et al.: Bayesian nonparametric federated learning of neural networks. In: Proceedings of ICML, pp. 7252–7261 (2019)

    Google Scholar 

  43. Zhang, F., Li, Q., Ren, Y., Xu, H., Song, Y., Liu, S.: An expression recognition method on robots based on mobilenet v2-ssd. In: Proceedings of ICSAI, pp. 118–122 (2019)

    Google Scholar 

Download references

Acknowledgements

The authors extend their sincere gratitude to Prof Hongbo Liu from the Dalian University of Technology, China for the useful discussions. Dr Mufti Mahmud is supported by the AI-TOP (2020-1-UK01-KA201-079167) and DIVERSASIA (618615-EPP-1-2020-1-UKEPPKA2-CBHEJP) projects funded by the European Commission under the Erasmus+ programme.

Author information

Authors and Affiliations

  1. United International University, Dhaka, Bangladesh

    Tapotosh Ghosh

  2. Bangladesh University of Professionals, Dhaka, Bangladesh

    Md. Hasan Al Banna & Md. Jaber Al Nahian

  3. Jahangirnagar University, Savar, Dhaka, Bangladesh

    M. Shamim Kaiser

  4. Department of Computer Science, Nottingham Trent University, Clifton Campus, Nottingham, NG11 8NS, UK

    Mufti Mahmud

  5. MTIF and CIRC, Nottingham Trent University, Clifton Campus, Nottingham, NG11 8NS, UK

    Mufti Mahmud

  6. School of Electrical Engineering, Yanshan University, Qinhuangdao, 066004, China

    Shaobao Li

  7. Department of Computer Science, University of Pretoria, Hillcrest, Pretoria, South Africa

    Nelishia Pillay

Authors
  1. Tapotosh Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Md. Hasan Al Banna
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Md. Jaber Al Nahian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Shamim Kaiser
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mufti Mahmud
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shaobao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nelishia Pillay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tapotosh Ghosh .

Editor information

Editors and Affiliations

  1. Nottingham Trent University, Nottingham, UK

    Mufti Mahmud

  2. University Mediterranea of Reggio Calabria, Reggio Calabria, Italy

    Cosimo Ieracitano

  3. Jahangirnagar University, Dhaka, Bangladesh

    M. Shamim Kaiser

  4. University Mediterranea of Reggio Calabria, Reggio Calabria, Italy

    Nadia Mammone

  5. University Mediterranea of Reggio Calabria, Reggio Calabria, Italy

    Francesco Carlo Morabito

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper

Ghosh, T. et al. (2022). A Privacy-Preserving Federated-MobileNet for Facial Expression Detection from Images. In: Mahmud, M., Ieracitano, C., Kaiser, M.S., Mammone, N., Morabito, F.C. (eds) Applied Intelligence and Informatics. AII 2022. Communications in Computer and Information Science, vol 1724. Springer, Cham. https://doi.org/10.1007/978-3-031-24801-6_20

Download citation

Publish with us

Policies and ethics

Search

Navigation