Skip to main content
SpringerLink
Log in
Book cover

Annual Conference of the Japanese Society for Artificial Intelligence

JSAI 2020: Advances in Artificial Intelligence pp 239–247Cite as

New Approach to Estimating Mental Health Score Using a Communication Agent

  • Conference paper
  • First Online:

  • 298 Accesses

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1357))

Abstract

It is important for monitoring systems for the elderly to estimate the state of the user during an interaction. In the field of elderly welfare, quality of life (QOL) is a useful indicator for comprehensive treatment of physical suffering as well as mental health and social well-being. In this study, we propose a new approach to QOL estimation that integrates facial expressions, head fluctuations, and eye movements captured during interactions with a communication agent. To this end, we implemented a communication agent and constructed a database based on the information collected through interpersonal experiments with 14 participants. In addition, we implemented a multimodal learning estimator and performed training with facial expressions, head fluctuation, and gaze feature extraction. Our results show that integration learning realized with all of the features integrated provided less error than single-modal learning based on facial expression alone. From our experimental results, we concluded that the proposed system can be used adequately as a QOL estimator.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Cabinet Office (CAO): Annual report on the ageing society 2018 (2018). https://www8.cao.go.jp/kourei/english/annualreport/2018/2018pdf_e.html

  2. Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: Realtime multiperson 2D pose estimation using part affinity fields. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302–1310 (2017). https://doi.org/10.1109/CVPR.2017.143

  3. Cohn, J.F., Reed, L.I., Ambadar, Z. Xiao, J., Moriyama, T.: Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. In: 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), vol. 1, pp. 610–616. IEEE (2004). https://doi.org/10.1109/ICSMC.2004.1398367

  4. DeVault, D., et al.: SimsenSei Kiosk: a virtual human interviewer for healthcare decision support. In: Proceedings of the 2014 international Conference on Autonomous Agents and Multi-agent Systems, pp. 1061–1068. International Foundation for Autonomous Agents and Multiagent Systems (2014)

    Google Scholar 

  5. docomo developer support, NTT Docomo. (n.d.). https://dev.smt.docomo.ne.jp. Accessed 26 Feb 2020

  6. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 445–450 (2016). https://doi.org/10.1145/2993148.2997632

  7. Fukuhara, S., Bito, S., Green, J., Hsiao, A., Kurokawa, K.: Translation, adaptation, and validation of the SF-36 health survey for use in Japan. J. Clin. Epidemiol. 51(11), 1037–1044 (1998). https://doi.org/10.1016/S0895-4356(98)00095-X

    Article  Google Scholar 

  8. Higuchi, M., et al.: Discrimination of bipolar disorders using voice. In: Cipresso, P., Serino, S., Villani, D. (eds.) MindCare 2019. LNICSSITE, vol. 288, pp. 199–207. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25872-6_16

    Chapter  Google Scholar 

  9. Hu, M., Yang, C., Zheng, Y., Wang, X., He, L., Ren, F.: Facial expression recognition based on fusion features of center-symmetric local signal magnitude pattern. IEEE Access 7, 118435–118445 (2019)

    Article  Google Scholar 

  10. Kocaguneli, E., Menzies, T.: Software effort models should be assessed via leave-one-out validation. J. Syst. Softw. 86(7), 1879–1890 (2013). https://doi.org/10.1016/j.jss.2013.02.053

    Article  Google Scholar 

  11. Liu, P., Yin, L.: Spontaneous facial expression analysis based on temperature changes and head motions. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–6. IEEE (2015). https://doi.org/10.1109/FG.2015.7163094

  12. Lu, Y., Zheng, W.L., Li, B., Lu, B.L.: Combining eye movements and EEG to enhance emotion recognition. In: Proceedings of the 24th International Conference on Artificial Intelligence (IJCAI), pp. 1170–1176. AAAI Press (2015)

    Google Scholar 

  13. Mori, T., et al.: Life pattern estimation of the elderly based on accumulated activity data and its application to anomaly detection. J. Robot. Mechatron. 24(5), 754–765 (2012). https://doi.org/10.20965/jrm.2012.p0754

    Article  Google Scholar 

  14. Nakagawa, S., Enomoto, D., Yonekura, S., Kanazawa, H., Kuniyoshi, Y.: A telecare system that estimates quality of life through communication. In: 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), pp. 325–330 (2018). https://doi.org/10.1109/CCIS.2018.8691360

  15. Nakagawa, S., Enomoto, D., Yonekura, S., Kanazawa, H., Kuniyoshi, Y.: New telecare approach based on 3D convolutional neural network for estimating quality of life. Neurocomputing 397, 464–476 (2020). https://doi.org/10.1016/j.neucom.2019.09.112

    Article  Google Scholar 

  16. Ren, F., Dong, Y., Wang, W.: Emotion recognition based on physiological signals using brain asymmetry index and echo state network. Neural Comput. Appl. 1–11 (2018). https://doi.org/10.1007/s00521-018-3664-1

  17. Ren, F., Kang, X., Quan, C.: Examining accumulated emotional traits in suicide blogs with an emotion topic model. IEEE J. Biomed. Health Inform. 20(5), 1384–1396 (2015). https://doi.org/10.1109/JBHI.2015.2459683

    Article  Google Scholar 

  18. Ren, F., Wu, Y.: Predicting user-topic opinions in Twitter with social and topical context. IEEE Trans. Affect. Comput. 4(4), 412–424 (2013). https://doi.org/10.1109/T-AFFC.2013.22

    Article  Google Scholar 

  19. Solbach, M.D., Tsotsos, J.K. Vision-based fallen person detection for the elderly. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1433–1442 (2017). https://doi.org/10.1109/ICCVW.2017.170

  20. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497 (2015). https://doi.org/10.1109/ICCV.2015.510

  21. The United Nations: World population ageing 2019 (2019). https://www.un.org/en/development/desa/population/publications/pdf/ageing/WorldPopulationAgeing2019-Report.pdf

  22. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p. I (2001). https://doi.org/10.1109/CVPR.2001.990517

  23. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009). https://doi.org/10.1109/TPAMI.2008.52

    Article  Google Scholar 

  24. Zhang, Y., Ji, Q.: Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 699–714 (2005). https://doi.org/10.1109/TPAMI.2005.93

    Article  Google Scholar 

  25. Zhao, L.M., Li, R., Zheng, W.L., Lu, B.L.: Classification of five emotions from EEG and eye movement signals: complementary representation properties. In: 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 611–614 (2019). https://doi.org/10.1109/NER.2019.8717055

Download references

Acknowledgements

This research was supported by the KDDI Foundation Research Grant Program 2019 and Graduate Program for Social ICT Global Creative Leaders (GCL) of The University of Tokyo by MEXT (Ministry of Education, Culture, Sports, Science and Technology).

Author information

Authors and Affiliations

  1. The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan

    Satoshi Nakagawa, Shogo Yonekura, Hoshinori Kanazawa, Satoshi Nishikawa & Yasuo Kuniyoshi

Authors
  1. Satoshi Nakagawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shogo Yonekura
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hoshinori Kanazawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Satoshi Nishikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yasuo Kuniyoshi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Satoshi Nakagawa .

Editor information

Editors and Affiliations

  1. Kansai University, Suita, Osaka, Japan

    Katsutoshi Yada

  2. Department of Applied Computer Science, Tokyo Polytechnic University, Atsugi, Kanagawa, Japan

    Daisuke Katagami

  3. Graduate School of System Design, Tokyo Metropolitan University, Hino, Tokyo, Japan

    Yasufumi Takama

  4. Department of Social Informatics, Kyoto University, Kyoto, Japan

    Takayuki Ito

  5. Division of Behavioral Science, Faculty of Letters, Chiba University, Chiba, Chiba, Japan

    Akinori Abe

  6. Department of Computer Science, Graduate School of System Design, Tokyo Metropolitan University, Hino, Tokyo, Japan

    Eri Sato-Shimokawara

  7. Mathematics and Informatics Center, The University of Tokyo, Tokyo, Japan

    Junichiro Mori

  8. Graduate School of Economics, Osaka University, Toyonaka, Osaka, Japan

    Naohiro Matsumura

  9. Department of Intelligence Science and Technology, Kyoto University, Kyoto, Japan

    Hisashi Kashima

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nakagawa, S., Yonekura, S., Kanazawa, H., Nishikawa, S., Kuniyoshi, Y. (2021). New Approach to Estimating Mental Health Score Using a Communication Agent. In: Yada, K., et al. Advances in Artificial Intelligence. JSAI 2020. Advances in Intelligent Systems and Computing, vol 1357. Springer, Cham. https://doi.org/10.1007/978-3-030-73113-7_24

Download citation

Publish with us

Policies and ethics

Search

Navigation