Skip to main content
SpringerLink
Log in

Baseline-independent stress classification based on facial StO2

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Stress is an affective human state. Different types of stress have varying effects on human health. Previous stress classifications using tissue oxygen saturation (StO2) have relied on baselines to extract features in a way that limits practical application since the baselines cannot be preacquired. In this paper, we explored the correlation and the classification computability of the baseline-independent facial StO2 across four states involving baseline, emotional stress, high-intensity physical stress and low-intensity physical stress. Three analytical approaches verify that facial StO2 is affection-related and state-specific. Building on this theoretical foundation, we validated the baseline-independent facial StO2 before and after modified data augmentation using three input forms as well as two network structures. We also proposed a dual-stream (Channel stream and Spatial stream) attention network, named as CSnet, to perform baseline-independent stress classification. Experimental results suggest that the proposed method can achieve a unweighted average recall (UAR) of 0.6935 and unweighted F1-score (UF1) of 0.6991, which is higher than traditional image descriptors methods, and is more competitive than the baseline-dependent classification with respect of real applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The original StO2 stress database used in this paper was derived from our previous work (see [8]), which was publicly released in 2020. In this paper, we further provide the mixup-expanded StO2 matrix data and RGB image format (not included in original database [8]) corresponding to all existing StO2 to comprise the database version II. Researchers can access the database by contacting the corresponding author to sign the License Agreement.

References

  1. Resendiz-Ochoa E, Cruz-Albarran IA, Garduño-Ramon MA, Rodriguez-Medina DA, Osornio-Rios RA, Morales-Hernández LA (2021) Novel expert system to study human stress based on thermographic images. Exp Syst Appl 178:115024. https://doi.org/10.1016/j.eswa.2021.115024

    Article  Google Scholar 

  2. Arsalan A, Majid M (2021) Human stress classification during public speaking using physiological signals. Comput Biol Med 133:104377

    Article  Google Scholar 

  3. Zafar MS, Nauman M, Nauman H, Nauman S, Kabir A, Shahid Z, Fatima A, Batool M (2021) Impact of stress on human body: a review. Eur J Med Health Sci 3(3):1–7. https://doi.org/10.1146/annurev.clinpsy.1.102803.144141

    Article  Google Scholar 

  4. Meaney MJ (2023) Chapter 42 - stress. In: Neurobiology of brain disorders (second edition), Academic Press, pp 781–791

  5. Dutcher JM, Creswell JD (2018) The role of brain reward pathways in stress resilience and health. Neurosci Biobehav Rev 95:559–567. https://doi.org/10.1016/j.neubiorev.2018.10.014

    Article  Google Scholar 

  6. Sara JDS, Toya T, Ahmad A, Clark MM, Gilliam WP, Lerman LO, Lerman A (2022) Mental stress and its effects on vascular health. Mayo Clin Proc 97 (5):951–990. https://doi.org/10.1016/j.mayocp.2022.02.004

    Article  Google Scholar 

  7. Madison A, Kiecolt-Glaser JK (2019) Stress, depression, diet, and the gut microbiota: human–bacteria interactions at the core of psychoneuroimmunology and nutrition. Curr Opin Behav Sci 28:105–110. https://doi.org/10.1016/j.cobeha.2019.01.011

    Article  Google Scholar 

  8. Liu XY, Shan YH, Peng M, Chen HY, Chen T (2020) Human stress and sto2: database, features, and classification of emotional and physical stress. Entropy (Basel, Switzerland) 22(9):962

    Article  Google Scholar 

  9. Sriramprakash S, Prasanna VD, Murthy OVR (2017) Stress detection in working people. Proc Comput Sci 115:359–366

    Article  Google Scholar 

  10. Moridani MK, Mahabadi Z, Javadi N (2020) Heart rate variability features for different stress classification. Bratislavské lé,karské listy 121(9):619–627

    Google Scholar 

  11. Sharma LD, Bohat VK, Habib M, Al-Zoubi AM, Faris H, Aljarah I (2022) Evolutionary inspired approach for mental stress detection using eeg signal. Exp Syst Appl 197:116634

    Article  Google Scholar 

  12. Machado F., José R, Anishchenko L (2018) Mental stress detection using bioradar respiratory signals. Biomed Signal Process Control 43:1746–8094

    Google Scholar 

  13. Shan YH, Li SG, Chen T (2020) Respiratory signal and human stress: non-contact detection of stress with a low-cost depth sensing camera. Int J Mach Learn Cybern 11(8):1825–1837. https://doi.org/10.1007/s13042-020-01074-x

    Article  Google Scholar 

  14. Hong K, Liu GD, Chen WT, Hong S (2018) Classification of the emotional stress and physical stress using signal... Pattern Recog 77:140–149. https://doi.org/10.1016/j.patcog.2017.12.013

    Article  Google Scholar 

  15. Jaroslaw J, Agata F, Winiarska A, Kaczmarczyk M, Poniatowski A (2019) Cardiovascular response to different types of acute stress stimulations. Folia Med Cracov 59(4):95–110. https://doi.org/10.24425/fmc.2019.131383

    Google Scholar 

  16. Chen T, Yuen PWT, Hong K, Ibrahim I, Tsitiridis A, Soori U, Jackman J, James D, Richardson M (2011) Assessment of tissue blood perfusion in-vitro using hyperspectral and thermal imaging techniques. Paper presented at 5th international conference on bioinformatics and biomedical, May 2011. https://doi.org/10.1109/icbbe.2011.5780189

  17. Chen T, Yuen P, Hong K, Tsitiridis A, Kam F, Jackman J, James D, Richardson M, Oxford W, Piper J, Thomas F, Lightman S Remote sensing of stress using electro-optics imaging technique

  18. Chen T, Yuen P, Richardson M, Liu GY, She ZS (2014) Detection of psychological stress using a hyperspectral imaging technique. IEEE Trans Affect Comput 5:391–405. https://doi.org/10.1109/TAFFC.2014.2362513

    Article  Google Scholar 

  19. Chen T, Yuen P, Richardson M, She Z, Liu G (2015) Wavelength and model selection for hyperspectral imaging of tissue oxygen saturation. Imaging Sci J 63 (5):290–295. https://doi.org/10.1179/1743131X15Y.0000000007

    Article  Google Scholar 

  20. Hong K, Liu XL, Liu GD, Chen WT (2019) Detection of physical stress using multispectral imaging. Neurocomputing 329:116–128

    Article  Google Scholar 

  21. Liu XY, Xiao X, Cao RL, Chen T (2020) Evolution of facial tissue oxygen saturation and detection of human physical stress. Paper presented at Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), 2020. https://doi.org/10.1109/IPEC49694.2020.9115140

  22. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Sci (Am Assoc Adv Sci) 313(5786):504–507

    Article  MathSciNet  MATH  Google Scholar 

  23. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, pp 3856–3866

  24. Nummenmaa L, Glerean E, Hari R, Hietanen JK (2014) Bodily maps of emotions. Proc Natl Acad Sci (PNAS) 111(2):646– 651

    Article  Google Scholar 

  25. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6(1):1–48

    Article  Google Scholar 

  26. Zhang YJ, Liu WJ, Fan HS, Zou YJ, Cui ZW, Wang Q (2021) Dictionary learning and face recognition based on sample expansion. Appl Intell 52(4):3766–3780

    Article  Google Scholar 

  27. Zhang HY, Ciss M, Dauphin YN, Lopez-paz D (2018) Mixup: beyond empirical risk minimization. In: 6Th international conference on learning representations, ICLR 2018. OpenReview

  28. Jia D, Wei D, Socher R, Li JL, Kai L, Li FF (2009) ImageNet: A large-scale hierarchical image database. Paper presented at IEEE conference on computer vision and pattern recognition, 2009. https://doi.org/10.1109/CVPR.2009.5206848

  29. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Handbook Systemic Autoimmune Diseases 1(4):

  30. Li DH, Zhao XC, Yuan GJ, Liu Y, Liu GY (2020) Robustness comparison between the capsule network and the convolutional network for facial expression recognition. Appl Intell 51(4):2269–2278. https://doi.org/10.1007/s10489-020-01895-x

    Article  Google Scholar 

  31. Tian C, Yuan Y, Zhang S, Lin C-W, Zuo W, Zhang D (2022) Image super-resolution with an enhanced group convolutional neural network. Neural Netw 153:373–385. https://doi.org/10.1016/j.neunet.2022.06.009

    Article  Google Scholar 

  32. Yun WH, Lee DJ, Park C, Kim J, Kim J (2020) Automatic recognition of children engagement from facial video using convolutional neural networks. IEEE Trans Affect Comput 11(4):696–707

    Article  Google Scholar 

  33. Song TF, Zheng WM, Song P, Cui Z (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541

    Article  Google Scholar 

  34. Liu Y, Ji LX, Huang RY, Ming TSY, Gao C, Zhang JP (2020) An attention-gated convolutional neural network for sentence classification. Intell Data Anal 23(5):1091–1107. https://doi.org/10.3233/IDA-184311

    Article  Google Scholar 

  35. Lucey P, Cohn JF, Kanade T, Saragih JM, Ambadar Z, Matthews IA (2010) The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE conference on computer vision and pattern recognition, CVPR workshops 2010, pp 94–101. https://doi.org/10.1109/CVPRW.2010.5543262

  36. Liu YC, Du HM, Zheng L, Gedeon T (2019) A Neural Micro-Expression Recognizer. Paper presented at 14th IEEE international conference on automatic face and gesture, 2019. https://doi.org/10.1109/FG.2019.8756583

  37. Woo S, Park J, Lee J, Kweon IS (2018) Cbam: Convolutional block attention module. In: Computer vision - ECCV 2018 - 15th european conference, Munich, Germany, vol 11211. pp 3–19. https://doi.org/10.1007/978-3-030-01234-2_1

  38. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: IEEE conference on computer vision and pattern recognition(CVPR), pp 3146–3154. https://doi.org/10.1109/CVPR.2019.00326

  39. Sagar A (2022) Dmsanet: Dual multi scale attention network. In: Image analysis and processing - ICIAP 2022 - 21st international conference, vol 13231. pp 633–645

  40. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24:971–987

    Article  MATH  Google Scholar 

  41. Zhou SR, Yin JP, Zhang JM (2013) Local binary pattern (lbp) and local phase quantization (lbq) based on gabor filter for face representation. Neurocomputing 116:260–264

    Article  Google Scholar 

  42. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition(CVPR), pp 886–893. https://doi.org/10.1109/CVPR.2005.177

  43. Bosch A, Zisserman A, Muñoz X (2007) Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM international conference on image and video retrieval(CIVR), pp 401–408. https://doi.org/10.1145/1282280.1282340

Download references

Author information

Authors and Affiliations

  1. College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China

    Xinyu Liu, Dong Chen, Ju Zhou & Tong Chen

  2. Chongqing Key Laboratory of Non-linear Circuit and Intelligent Information Processing, Southwest University, Chongqing, 400715, China

    Xinyu Liu, Dong Chen, Ju Zhou & Tong Chen

  3. Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing, 400715, China

    Xinyu Liu, Dong Chen, Ju Zhou & Tong Chen

Authors
  1. Xinyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ju Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tong Chen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Dong Chen and Ju Zhou are contributed equally to this work.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Chen, D., Zhou, J. et al. Baseline-independent stress classification based on facial StO2. Appl Intell 53, 10255–10272 (2023). https://doi.org/10.1007/s10489-022-04041-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04041-x

Keywords

Search

Navigation